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EXECUTIVE  SUMMARY 


Research  Requirement: 

Military  leaders  are  taking  on  new  and  varied  missions.  The  amount  of  infonnation  required  to 
maintain  competence  is  sometimes  overwhelming  and  cannot  be  learned  during  class  time  in  a 
school  house.  The  requirement  for  a  self-paced,  scenario-based  training  tool  that  can  be  updated 
to  maintain  relevance  on  new  mission  sets  provided  the  stimulus  for  this  research. 

Procedure: 

Guided  by  the  training  requirements  and  design  concept,  a  set  of  computer-based  training 
tutorials  called  the  Stability  Operations  in  Joint,  Interagency,  Intergovernmental,  and 
Multinational  (JIIM)  Environments  Tutor  was  developed.  Three  versions  of  the  tutorials  were 
developed  iteratively.  Each  version  was  pilot-tested  by  students  attending  the  Advanced  Stability 
Operations  course  at  the  U.S.  Anny  Command  and  General  Staff  College  (CGSC).  After  each 
version  was  used  by  the  class,  feedback  was  gathered  from  students  and  instructors  to  identify 
improvements  which  were  incorporated  in  the  next  version  of  the  tutorials.  The  tutorials  include: 

•Cross-Cultural  Competence  Inventory 
•Introduction  to  Operations  in  JIIM  Environments 
•Interagency  Management  System  in  Action! 

•  Security  in  Sudan 
•Governance  and  Justice  in  Haiti 

•Humanitarian  Assistance  and  Economic  Development  in  Afghanistan 

Traditional  training  systems  present  information  and  then  test  their  recall.  By  contrast,  the 
tutorials  interweave  the  learning  of  FM  3-07  and  other  doctrine  with  problems  and  scenarios. 
These  scenarios  challenge  students  to  apply  their  knowledge  and  cognitive  skills  to  assess 
situations  and  generate  and  evaluate  options  in  representative  situations.  This  training  approach 
accelerates  learning  by  providing  scenario-based  learning  goals  and  contexts  for  acquiring, 
integrating,  and  retaining  declarative  knowledge  more  effectively.  The  scenarios  also  provide 
practice  opportunities  for  applying  high-level  cognitive  skills  described  by  the  JIIM  themes. 

To  enable  instructional  developers  to  create  and  maintain  these  types  of  training  systems, 
an  authoring  tool  was  developed  that  was  built  on  top  of  an  existing  technology.  Like  other 
authoring  tools  for  computer-based  training,  this  tool  enables  authors  to  create  a  sequence  of 
screens  that  present  information  using  formatted  text  and  graphics,  pose  questions,  and  provide 
feedback.  In  addition,  the  authoring  tool  simplifies  the  inclusion  of  logic  that  selects  and  presents 
information,  hints,  and  feedback  adaptively,  based  upon  the  student’s  previous  answers. 
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Findings: 


Training  effectiveness  was  indicated  by  evaluating  the  tutorial  that  taught  the  Interagency 
Management  System  by  analyzing  pre-  and  post-test  scores  and  participant  responses.  Participant 
responses  were  positive,  however  negative  responses  indicated  that  “the  tutorial  took  too  long  to 
complete.” 

In  addition  to  its  utility  as  a  training  system,  the  tutorials  served  as  a  proof  of  concept  that 
illustrates  the  feasibility  and  effectiveness  of  this  type  of  computer-based  instruction  for 
accelerating  introductory  learning.  It  also  served  as  a  pilot  system  that  was  analyzed  to  identify 
ways  of  improving  its  instruction.  Experience  developing  the  tutorials  also  guided  the 
development  of  the  authoring  tool  for  creating  this  type  of  instruction  more  easily.  The  authoring 
tool  was  reviewed  by  a  fonner  instructor  at  Joint  Forces  Staff  College  who  concluded  that  the 
tool  could  create  useful  instructional  content  and  could  be  used  effectively  by  instructors. 

Utilization  and  Dissemination  of  Findings: 

The  tutorials  have  been  selected  for  use  by  the  Joint,  Interagency,  and  Multinational 
Planner's  Course  (JIMPC),  offered  at  Joint  Forces  Staff  College  and  the  Advanced  Stability 
Operations  Course  at  CGSC.  The  Office  of  the  Secretary  of  State  is  using  portions  of  the  tutorial 
to  train  civil-military  collaboration.  The  IMS  Tutorial  has  been  reviewed  by  members  of  the 
Office  of  the  Secretary  of  Defense  (OSD)  Personnel  and  Readiness.  It  is  under  consideration  by 
US  JFCOM  to  educate  Combatant  Commander  and  Joint  Task  Force  staffs  on  the  Interagency 
Management  System  who  cannot  attend  the  S/CRS  Foundations  course  and  must  rely  on  distance 
learning  methods.  The  tutorials  are  also  being  evaluated  for  wider  use  by  other  schools  at  Joint 
Forces  Staff  College  and  by  the  Swedish  Defence  Research  Agency  as  a  tool  for  collaboration  in 
Stability  Operations. 
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TRAINING  JOINT,  INTERAGENCY,  INTERGOVERNMENTAL,  AND 
MULTINATIONAL  (JIIM)  PARTICIPANTS  FOR  STABILITY  OPERATIONS 

INTRODUCTION 

The  operational  environment  for  U.S.  military  forces  requires  the  comprehension  of 
significant  amounts  of  new  infonnation  to  gain  deep  understanding  of  problem  sets.  Training 
must  be  relevant  and  should  incorporate  context  for  the  new  knowledge  to  be  gained  rapidly  and 
retained  well.  Scenario  based  training  can  accelerate  knowledge  acquisition  and  retention. 

The  purpose  of  this  project  was  to  support  the  training  and  education  process  that 
prepares  U.S.  forces  and  their  potential  partners  to  plan  and  implement  Stability  Operations  in 
Joint,  Interagency,  Intergovernmental,  and  Multinational  (JIIM)  environments. 

The  approach  taken  started  with  analyzing  requirements  for  understanding  Stability 
Operations  in  the  JIIM  environment  and  understanding  the  target  audience.  A  set  of  themes  was 
developed  that  reflect  the  high-level  cognitive  skills  that  experts  use  to  be  successful  in  JIIM 
environments.  Documentation  of  the  themes  can  be  found  in  Ross,  Grome,  Arrastia,  Schaab, 

Ong,  and  Spangler  report  published  by  the  U.S.  Army  Research  Institute  in  2010. 

Guided  by  the  training  requirements  and  training  design  concept,  a  collection  of 
computer-based  training  tutorials  called  the  Stability  Operations  in  Joint,  Interagency, 
Intergovernmental,  and  Multinational  (JIIM)  Environments  Tutor  was  developed.  Three  versions 
of  the  tutorials  were  developed  iteratively.  Training  content  was  expanded  and  refined  in  each 
new  version.  Each  version  was  pilot-tested  by  students  attending  the  Advanced  Stability 
Operations  course  at  the  U.S.  Army  Command  and  General  Staff  College  (CGSC).  After  each 
version  of  the  tutorial  was  used  by  the  class,  feedback  was  gathered  from  students  and  instructors 
to  identify  additional  improvements  which  were  incorporated  in  the  next  version. 

Traditional  training  systems  present  infonnation  and  then  test  their  recall.  By  contrast,  the 
tutorials  interweave  the  learning  of  FM  3-07  and  other  doctrine  with  problems  and  scenarios. 
These  scenarios  challenge  students  to  apply  their  knowledge  and  cognitive  skills  to  assess 
situations  and  generate  and  evaluate  options  in  representative  situations.  This  training  approach 
accelerates  learning  by  providing  scenario-based  learning  goals  and  contexts  for  acquiring, 
integrating,  and  retaining  declarative  knowledge  more  effectively  (Ross  &  Lussier,  1999;  Schaab 
&  Dressel,  2001).  The  scenarios  also  provide  practice  opportunities  for  applying  high-level 
cognitive  skills  described  by  the  JIIM  themes. 

Training  effectiveness  of  the  tutorial  that  taught  the  Interagency  Management  System 
(IMS)  was  evaluated  by  analyzing  pre-  and  post-test  scores  and  student  responses  to  a  reaction 
survey.  To  enable  instructors  and  subject  matter  experts  to  create  and  maintain  these  types  of 
training  systems,  an  authoring  tool  was  developed  that  was  built  on  top  of  an  existing 
technology. 
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METHODS 

Domain  Analysis 

A  number  of  JIIM-related  documents  were  reviewed  to  understand  the  issues,  challenges, 
and  lessons  learned  about  operating  in  this  environment.  Findings  were  documented  in  a  lessons 
learned  report  by  Agrait  and  Loughran  (2007).  Twenty-two  additional  documents  were  reviewed 
which  resulted  in  extraction  of  approximately  400  items  of  domain  knowledge  comprised  of 
facts,  concepts,  skills,  guidelines  and  heuristics,  tasks,  and  challenging  situations  that  relate  to 
JIIM  stability  and  support  operations.  This  background  infonnation  provided  training 
requirements  and  provided  details  which  could  be  included  in  training  scenarios. 

During  the  project,  doctrine,  processes,  organizations,  and  field  experiences  in  the  area  of 
Stability  Operations  evolved  rapidly,  and  they  continue  to  evolve.  In  particular,  the  new  Stability 
Operations  (2008)  (Army  Field  Manual  3-07)  was  published,  and  the  Interagency  Management 
System  (IMS)  was  evolving  under  the  direction  of  the  State  Department’s  Office  of  the 
Coordinator  for  Reconstruction  and  Stabilization  (S/CRS).  This  manual  defines  Stability 
Operations  more  clearly  than  any  prior  document,  and  it  embeds  them  in  Comprehensive 
Approach  that  calls  for  regular  coordination,  consultation,  and  interaction  among  all  actors 
involved  including  the  range  of  JIIM  participants.  The  manual  is  organized  around  five  stability 
tasks  to  integrate  activities  at  the  tactical  level  (see  Figurel): 


1 .  Establish  Civil  Security, 

2.  Establish  Civil  Control, 

3.  Restore  Essential  Services, 

4.  Support  to  Governance  and 

5.  Support  to  Economic  and  Infrastructure  Development. 

Five  tasks  support  the  Stability  Sectors  to  create  unity  of  effort  and  a  vision  of  the  desired 
end  state  across  all  participants  in  the  JIIM  environment.  The  sectors  are: 

1 .  Security, 

2.  Justice  and  Reconciliation, 

3.  Humanitarian  Assistance  and  Social  Well-Being, 

4.  Governance  and  Participation,  and 

5.  Economic  Stabilization  and  Infrastructure. 

Development  of  Training  Themes 

People  with  experience  in  some  aspect  of  JIIM  environments  were  interviewed.  The  goal 
was  to  compare  multiple  perspectives  at  tactical  and  operations  levels,  particularly  given  the 
attention  that  has  been  focused  on  how  tactical  interactions  have  operational  and  strategic 
impacts  in  modem  operations  (see  for  example,  Krulak,  1999).  Participants  in  the  military  or 
associated  with  military  organizations  were  volunteers,  and  participants  from  other  organizations 
were  compensated  for  their  time  as  subject  matter  experts.  Twenty-five  people  were  interviewed: 

•  Seventeen  military  or  military-related  Subject  Matter  Experts  (SMEs)  (one  of  which  was 
not  used  for  analysis) 

•Three  African  nationals,  one  affiliated  with  a  non-governmental  organization  and  two  with  a 
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government  health  organization 
•Five  SMEs  with  Department  of  State  experience 


STABILITY  TASKS 


STABILITY  SECTORS 


Figure  1.  Stability  Tasks  Linked  to  Stability  Sectors  (adapted  from  U.S.  Army  FM  3-07,  Stability 
Operations,  October  2008,  paragraph  2-20) 


A  semi-structured  interview  protocol  based  on  the  Critical  Decision  Method  (CDM; 
Crandall,  Klein,  &  Hoffman,  2006;  Hoffman,  Crandall,  &  Shadbolt,  1998)  was  used.  All 
interviews  were  recorded  and  transcribed  for  analysis.  The  interview  questions  were  developed 
to  help  indirectly  assess  the  elements  of  expertise  by  gathering  rich  examples  of  perfonnance  and 
insights  and  strategies  that  drove  perfonnance.  One  assumption  of  the  research  is  that  as 
expertise  develops,  people  generally  are  not  able  to  articulate  the  how  and  why  of  their  cognitive 
performance.  Therefore,  the  analysis  is  grounded  in  the  specifics  of  the  data,  but  the  data 
themselves  were  collected  to  expose  the  cognitive  challenges  and  strategies  inherent  in  situations 
without  directly  asking  participants  why  they  did  things  the  way  they  did. 

Design  Principles 

In  August  2008,  the  team  received  the  opportunity  to  work  with  the  U.S.  Army 
Command  and  General  Staff  College  (CGSC).  CGSC  had  developed  and  was  delivering  a  new 
course  called  Advanced  Stability  Operations  and  agreed  to  allow  us  to  use  students  in  the  class  to 
pilot-test  successive  versions  of  the  product.  This  relationship  helped  to  focus  the  training 
requirements. 
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The  challenge  was  figuring  out  how  to  integrate  foundational  knowledge  along  with 
insights  into  successful  collaboration  for  planning  and  implementation  for  a  diverse  training 
audience.  The  potential  distance  learning  audience  for  the  course  was  comprised  of  students  from 
different  Services,  different  countries,  and  different  government  agencies,  much  like  the 
composition  of  players  who  would  be  learning  and  perfonning  together  in  real  operations. 
Participants  in  the  course  were  projected  to  include  Army  officers  in  attendance  at  CGSC,  U.S. 
officers  from  other  U.S.  services,  foreign  officers  in  attendance  at  CGSC  or  currently  assigned  to 
other  U.S.  military  organizations,  U.S.  Government  civilians,  members  of  non-government 
organizations,  and  students  from  other  countries.  These  potential  learners  varied  widely  in  their 
knowledge  and  experiences  with  JIIM  operations. 

Based  on  the  training  requirements  analysis  described  above  and  the  subsequent  work 
with  instructors  at  CGSC,  the  training  system  focused  on  providing  students  with  an 
understanding  of: 

•the  foundation  knowledge  needed  to  understand  Stability  Operations, 

•the  high-level  skills  necessary  to  perform  successfully  in  JIIM  environments, 

•the  interplay  between  tactical  and  operational  plans  and  actions, 

•the  new  U.S.  Army  doctrine  for  Stability  Operations  (FM  3-07),  and 
•the  Interagency  Management  System  (IMS). 

The  high-level  objectives  for  the  training  were  to  enable  students  to: 

•Understand  the  whole  of  government  approach  and  the  comprehensive  approach  and  their 
challenges; 

•Understand  the  tasks,  issues,  and  challenges  in  the  five  sectors  of  Stability  Operations; 

•Gain  insight  into  the  high-level  cognitive  skills  that  experts  use  to  support  success  in  JIIM 
environments;  and 

•Connect  the  doctrine  and  procedures  to  specific  contexts  to  enable  successful  participation 
in  more  advanced  exercises  and  training. 

It  was  agreed  that  everyone  in  the  potential  audience  could  benefit  from  participating  in 
initial  training  to  develop  a  common  base  of  declarative  knowledge,  as  well  as  insights  into  how 
to  work  together.  The  tutorial  product  was  intended  to  provide  the  common  basis  that  would 
allow  learners  to  prepare  for  exercises  and  operations  and  be  ready  to  address  challenging  issues 
with  a  common  language  and  understanding  of  the  goals  of  Stability  Operations  and  the  IMS. 

Based  on  the  results  of  the  document  review,  interviews  with  SMEs,  and  the  themes 
derived  from  these  interviews,  the  following  were  developed:  a  general  training  concept,  a  set  of 
design  principles,  and  high-level  learning  objectives  to  serve  as  a  framework  for  development. 
Specific  learning  objectives  were  developed  later  for  each  portion  of  the  training  product.  The 
concept  for  this  product  was  individual  learning  based  on  blending  a  constructivist  approach  that 
situates  learning  in  a  context  with  a  Socratic  method  that  poses  questions  and  challenges  to 
encourage  learners  to  think  through  the  context.  By  “constructivist  approach”  we  mean  that  the 
learning  of  declarative  knowledge  and  concepts  was  situated  in  specific  contexts,  requiring  the 
students  to  use  the  knowledge  as  tools  to  solve  problems  and  answer  questions  within  those 
contexts.  The  choice  of  the  constructivist  approach  was  based  on  the  diversity  of  learning  levels 
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and  experience  in  the  training  audience,  the  need  to  accelerate  learning,  and  documented 
limitations  of  typical  introductory  training. 

The  goal  was  to  accelerate  entry-level  learning  on  terminology  and  operations  for 
students.  It  would  prepare  them  for  effective  participation  in  class  exercises  for  the  pilot 
audience  and  also  for  large-scale  exercises  with  a  wide  range  of  participants.  To  accelerate 
learning,  we: 


•embedded  declarative  knowledge  in  specific  contexts, 

•provided  principles  of  successful  practice  in  JIIM  environments  (themes),  and 
•presented  challenges  faced  during  operations  for  the  student  to  work  through  using  those 
themes. 

The  use  of  operational  issues  illustrated  the  high-level  cognitive  challenges  such  as 
organizational  and  cultural  clashes  that  can  occur  in  JIIM  environments.  In  general,  operational 
issues  were  presented  simultaneously  with  declarative  knowledge  in  context.  This  approach 
allowed  more  advanced  students  to  be  engaged  in  the  material  as  well.  The  presentation  of 
situations  from  multiple  perspectives  allowed  students  to  practice  the  expert  ability  of 
perspective  taking. 

The  following  design  principles  support  this  training  concept: 

1. To  support  the  less  experienced  learner,  the  cases  are  constrained,  but  rich  enough  to  avoid 

one  right  answer  or  only  one  exemplar  in  a  lesson  which  leads  to  brittle  performance. 
Specific  elements  of  the  context  were  provided  to  the  learner  that  supported  the  concepts 
to  be  learned.  For  example,  if  the  tutorial  focused  on  the  security  aspect  of  a  situation, 
richer  infonnation  about  the  context  would  be  provided  in  this  area.  If  humanitarian 
issues,  for  example,  were  also  part  of  the  situation,  only  enough  detail  to  support 
analyzing  the  situation  was  supplied.  For  students  who  were  willing  and  able  to  go 
further  into  detailed  context  of  the  situation,  additional  readings  were  provided. 

2.  Mental  model  development  is  supported  by  practicing  the  thought  processes  necessary  for 

success  in  the  domain  (i.e.,  recognizing  relevant  information,  organizing  infonnation, 
making  sense  of  situations,  and  using  information  to  solve  problems.)  Exercises  and 
questions  in  each  tutorial  required  students  to  analyze  situations  and  make  choices  that  a 
professional  in  the  setting  might  have  to  make,  although  with  more  guidance  as  to  the 
necessity  of  the  analysis  or  choice  and  more  structure  to  guide  the  process.  For  example, 
choosing  appropriate  agencies  to  partner  with  based  on  the  problem  required  situation 
analysis  and  the  action  of  choosing  partners  with  relevant  expertise. 

3.  Feedback  allowed  students  to  see  what  they  have  missed  in  situations  and  how  more  expert 

practitioners  view  the  situation,  as  well  as  the  consequences  of  concentrating  on  the 
wrong  aspects  of  a  situation.  Sample  expert  responses  were  provided  with  an  explanation 
of  why  the  response  suited  the  situation.  Feedback  was  provided  in  the  form  of  examples 
of  expert  answers  and  pre-  and  post-test.  This  enables  students  to  gauge  their 
understanding  by  comparing  their  thought  processes  to  others  and  by  performing  checks 
on  knowledge. 
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4.  Working  through  “messy”  situations  promotes  better  concept  integration  when  the  learner 

must  try  to  understand  a  situation  before  feedback  is  given.  Situations  did  not  clearly 
point  to  one  right  answer  and  required  thought  and  choices  among  a  range  of  alternatives 
including  some  that  were  workable,  but  perhaps  not  the  most  effective  possible  response. 

5.  The  student  focus  was  on  understanding  a  mission  in  order  to  integrate  concepts.  Support 

was  provided  in  the  fonn  of  advanced  organizers  such  as  learning  objectives  and 
highlighting  core  concepts.  Students  were  not  left  to  “wander”  aimlessly  through  a 
situation.  Instead,  they  were  supported  with  organizers  to  understand  what  concepts  were 
being  illustrated  and  practiced. 

Development  Process 

The  tutorials  were  developed  in  three  iterations.  Each  version  was  pilot-tested  during 
CGSC  Advanced  Stability  Operations  course  offerings  in  October  2008,  May  2009,  and 
November  2009.  The  following  table  summarizes  the  training  content  included  in  each  version: 

Table  1.  Summary  of  Training  Content  Included  in  Each  Version  of  the  Tutorials 


Version  1 

Oct  2008 

•  Security  in  Sudan 

•  Governance  and  Justice  in  Haiti 

•  Economic  Development  and  Humanitarian  Assistance  in  Afghanistan 

Version  2 

May  2009 

•  Introduction  to  Operations  in  JIIM  Environments 

•  Cross-cultural  Competency  Index 

•  Pre-  and  post- tests  and  situation  judgment  tests  added  to  the  Sudan 
tutorial 

Version  3 

Nov  2009 

•  Interagency  Management  System  in  Action! 

•  Pre-  and  post-tests  for  the  Sudan,  Haiti,  Afghanistan,  and  IMS  tutorials 

After  each  pilot  test,  quantitative  and  qualitative  reaction  survey  data  collected  from 
students  was  analyzed.  Additional  content  and  software  usability  improvements  were  identified 
and  incorporated  in  the  next  version  of  the  tutorials.  The  individual  tutorials  and  examples  of 
inherent  design  principles  are  described  in  the  Findings  and  Discussion  section. 

Evaluation  and  Review 


Participants 

The  research  included  18  individuals  ranging  from  E-8  to  0-6  who  participated  in  all 
components  of  this  research.  Five  participants  were  students  in  the  Advanced  Stability 
Operations  seminar  at  CGSC.  Additional  participants  with  relevant  military  backgrounds  were 
recruited  to  take  The  Interagency  Management  System  in  Action  tutorial  for  the  TEE.  These 
participants  came  from  all  services  (Air  Force,  Army,  US  Coast  Guard,  Navy,  and  Marines)  and 
multiple  countries  (United  States,  Kenya,  Slovenia,  Latvia,  Korea,  and  El  Salvador).  They 
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averaged  20.2  years  of  military  service  and  had  been  deployed  all  over  the  world  (Iraq, 
Afghanistan,  Bosnia,  Saudi  Arabia,  Hungary,  Georgia,  Korea,  Thailand,  Australia,  Turkey,  Italy, 
UAE,  Haiti,  Somalia,  Sudan,  Kosovo,  and  Kuwait). 

Materials 

The  Interagency  Management  System  in  Action  tutorial.  The  tutorial  was  designed  as  an 
introduction  to  the  IMS.  It  focused  on  the  following  areas: 

•Employing  the  IMS 

•Integration  of  the  IMS  with  a  Military  headquarters 
•Operating  as  a  JIIM  Team 
•Team  building  and  collaboration  issues 
•Case  Study  activities  and  discussion 

Knowledge  Tests.  A  knowledge  test  was  used  to  assess  student  declarative  knowledge 
and  the  application  of  that  knowledge  with  regard  to  this  tutorial.  Prior  to  the  start  of  the  tutorial, 
students  completed  a  2  5 -item  knowledge  test  to  obtain  a  baseline  of  their  existing  knowledge  of 
IMS,  JIIM,  and  collaboration  issues.  The  test  items  were  developed  based  on  the  tutorial’s 
learning  objectives.  After  completion  of  the  tutorial,  students  completed  the  same  25-item 
knowledge  test  asking  them  to  apply  knowledge  obtained  during  the  tutorial.  The  knowledge 
tests  and  answers  are  shown  at  Appendix  A. 

Participant  Reaction  Survey.  Participants  completed  a  Reaction  Survey  at  the  completion 
of  the  tutorial.  The  intent  of  the  survey  was  to  collect  participants’  opinions  regarding  how  well 
and  in  what  ways  the  tutorial  supported  stated  learning  objectives.  The  participants  also 
completed  items  regarding  demographic  infonnation  including  rank,  military  branch,  nationality, 
years  of  service,  and  previous  deployments  as  part  of  the  survey. 

Procedure 

Prior  to  data  collection,  the  purpose  of  the  training  effectiveness  evaluation  was  described 
to  participants.  Immediately  before  taking  the  pre-test,  participants  reviewed  a  two-page 
document  to  leam  how  to  operate  the  tutorial  to  progress  to  the  next  screen,  review  previous 
screens,  and  access  related  documents.  They  were  told  that  the  tutorial  required  approximately 
two  to  four  hours  to  run,  depending  on  the  individual. 

Participants  completed  a  pre-test  comprised  of  25  multiple  choice  questions.  Then,  they 
completed  the  tutorial.  Finally,  they  completed  a  post-test  comprised  of  the  same  multiple  choice 
questions  and  filled  out  a  Reaction  Survey.  The  results  of  this  training  effectiveness  evaluation 
are  reported  in  the  Findings  and  Discussion  section. 

In  addition  to  the  primary  testing  at  CGSC,  some  or  all  of  the  product  components  were 
reviewed  during  the  final  stage  of  development  by  organizations  that  were  considering  using  the 
tutorials.  These  included: 

•  instructors  and  students  at  Joint  Forces  Staff  College  and 

•members  of  the  Office  of  the  Secretary  of  Defense  (OSD)  Personnel  and  Readiness. 

This  department  focuses  on  Interagency  Training  integration  and  works  with  the 
S/CRS  in  the  development  of  their  civilian  deployable  capability.  Reviewers  from 
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OSD  had  previously  attended  the  two  week  S/CRS-sponsored  IMS  Foundations 
course,  and  they  reviewed  the  tutorial  Interagency  Management  System  in  Action! 


Authoring  Tool  Development  and  Usability  Assessment 

To  support  the  development  of  the  tutorials,  we  enhanced  an  existing  authoring  tool 
called  Task  Guide.  An  overview  of  this  tool  can  be  found  in  Appendix  B.  After  version  1  of  the 
tutorials  was  developed,  additional  capabilities  were  added  to  the  authoring  tool  to  simplify 
authoring.  These  included: 

•A  graphical  editor  for  specifying  fonnatted  text  and  input  controls  (e.g.,  text  fields,  check 
boxes)  using  a  third-party  commercial  software  component.  When  creating  version  1,  it 
was  necessary  to  enter  raw  Hypertext  Markup  Language  (HTML)  tags  to  specify  the 
fonnatting  and  input  controls. 

•  Simplified  support  for  authoring  multiple  choice  items  with  response-specific  hints  and 
feedback.  Using  this  support,  authors  could  specify  all  aspects  of  a  multiple  choice  item 
such  as  its  question  (prompt),  possible  options,  correct  options,  and  response-specific 
hints  and  feedback  for  each  correct  and  incorrect  option.  Once  this  infonnation  was 
entered,  the  author  could  quickly  insert  screens  that  pose  questions,  prompt  for  selections, 
and  provide  hints  and  feedback  at  the  desired  locations  in  the  tutorial. 

•Generation  and  conditional  inclusion  of  text  within  a  screen.  This  feature  enables  tutorial 
authors  to  design  screens  that  include  logic  that  generates  text  or  selects  among  alternate 
blocks  of  text  based  on  the  student’s  previous  interactions  and  other  information.  These 
features  enhance  Task  Guide’s  ability  to  support  adaptive  instruction. 

The  tool  was  modified  to  improve  the  student  experience  by  reducing  the  amount  of 
scripting  that  was  needed  to  author  certain  kinds  of  tutorial  logic  such  as  saving  each  student’s 
answers  and  scores  in  files.  The  following  documents  that  describe  how  to  operate  Task  Guide 
and  maintain  the  tutorials  were  written  or  revised: 

•Task  Guide  Users’  Guide 

•Task  Guide  Authors’  Guide 

•Maintaining  the  Stability  Operations  Tutorials 

To  assess  the  usability  of  the  authoring  tool,  a  former  instructor  at  Joint  Forces  Staff 
College  reviewed  the  Task  Guide  manuals.  Then,  during  a  WebEx  web  conference,  developers 
demonstrated  how  to  operate  the  authoring  tool.  The  instructor  then  applied  this  knowledge  to 
change  portions  of  an  existing  tutorial.  The  instructor  then  provided  feedback  by  completing  a 
written  questionnaire  (presented  in  Appendix  C). 

Tutorials 

This  section  describes  the  training  content  developed  and  included  in  the  three  versions 
of  the  tutorials. 

Tutorial  version  1  -  October  2008 

This  version  of  the  training  system  included  three  tutorials  that  illustrated  the  concepts  of 
the  five  stability  sectors  and  enabled  the  learner  to  work  at  integrating  the  concepts  in  context. 
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Several  questions  or  dilemmas  were  presented  to  the  learner  throughout  each  tutorial  to  engage 
them,  help  them  to  become  familiar  with  the  basic  doctrine,  and  support  practicing  the  thought 
patterns  needed  to  plan  and  conduct  operations.  Examples  of  expert  points  of  view  for  each 
challenge  were  available  once  the  student  had  an  opportunity  to  think  through  the  issues 
presented  by  the  tutorials.  The  tutorials  provided  quick  access  to  reference  materials  via 
hyperlinks  to  deepen  understanding  on  various  aspects  of  the  problem  or  the  context. 

In  each  tutorial,  a  general  setting  and  knowledge  of  the  type  of  task  were  provided,  followed  by 
presentation  of  a  specific  set  of  conditions  and  issues  for  the  learner  to  resolve.  Some  of  the  eight 
JIIM  themes  were  woven  into  each  tutorial  to  support  practice  of  these  high-level  skills.  For 
example,  the  learner  might  be  prompted  to  take  the  perspective  of  other  players  to  answer  a 
question. 


Security  in  Sudan. 


Stability  Operations  in  Joint, 
Interagency,  Intergovernmental,  and 
Multinational  (JIIM)  Environments  Tutor 


Tutorial  2 
Security  in  Sudan 


J) 

QOO 
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The  Sudan  situation  involved  the  United  Nations 
(UN)  experiencing  problems  enforcing  the 
Comprehensive  Peace  Agreement  in  2007.  The  tutorial 
was  designed  to  help  the  learner  understand  the  variety 
of  participants  in  this  type  of  operation,  understand  the 
effects  of  cultural  competence,  understand  the 
coordination  necessary,  make  assessments  of  the 
security  situation,  and  evaluate  the  different  participants 
along  several  dimensions. 

This  tutorial  addressed  theme  1  (Understand  the 
situation  within  its  historical,  regional,  and  cultural 
context)  and  theme  2  (Understand  the  other 
participants). 


Governance  and  justice  in  Haiti. 


Stability  Operations  in  Joint, 
Interagency,  Intergovernmental,  and 
Multinational  (JIIM)  Environments  Tutor 


Tutorial  3 

Governance  and 
Justice  in  Haiti  (1994) 
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The  Haiti  tutorial  was  based  on  an  intervention 
in  1994  and  required  the  learner  to  conduct  a  mission 
analysis  by  examining  tasks  from  multiple  perspectives 
and  their  differing  priorities  (Government  of  Haiti  led 
by  President  Aristide,  United  Nations,  and  the  U.S.), 
assessing  risks  and  resources,  and  assessing  stakeholder 
agendas. 

This  tutorial  addressed  JIIM  theme  3  (Shift 
perspective)  and  theme  4  (Establish  and  maintain 
common  ground). 
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Humanitarian  assistance  and  economic  development  in  Afghanistan. 


The  Afghanistan  tutorial  was  based  on  a 
hypothetical  natural  disaster  in  which  the  student 
coordinates  integration  of  civilian  reserve  personnel 
into  Provincial  Reconstruction  Teams  (PRTs)  while 
simultaneously  seeking  to  gain  information  about  threat 
activities  in  the  area. 

This  tutorial  addressed  theme  5  (Build  capability 
to  affect  the  situation). 


Tutorial  version  2  -  May  2009 

For  version  2,  two  activities  were  added: 

•Introduction  to  Operations  in  JIIM  Environments 
•Cross-Cultural  Competence  Inventory  (3 Cl) 


Stability  Operations  in  Joint,  Interagency, 
Intergovernmental,  and  Multinational 
(JIIM)  Environments  Tutor 

Tutorial  4  -  Humanitarian 
Assistance  and  Economic 
Development  in  Afghanistan 

1#  QOO  Oj 


Introduction  to  operations  in  JIIM  environments. 


Stability  Operations  in  Joint,  Interagency, 
Intergovernmental,  and  Multinational 
(JIIM)  Environments  Tutor 


Introduction  to  Operations  in 
JIIM  Environments 
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The  range  and  variety  of  potential  learners  was 
the  motivation  for  adding  the  Introduction  to 
Operations  in  JIIM  Environments  module.  The 
introduction  was  designed  to  insure  that  all  learners 
understood  the  wide  range  of  missions  and  tasks  that 
could  fall  under  Stability  Operations,  as  well  as  the 
challenges  and  issues  inherent  in  the  operations.  The 
introduction  provided  information  on  the  concepts  of 
the  Whole  of  Government  Approach  and  the 
Comprehensive  Approach,  as  well  as  describing  the 
Stability  Operations  tasks  to  insure  a  common 
understanding  of  the  basics  of  doctrine.  The  role  of  the 
US  military  as  an  actor  and  not  a  leader  in  many  situations  was  also  addressed. 
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Cross-cultural  competence  inventory  (3CI). 


The  Cross-Cultural  Competence  Inventory 
(3  Cl)  was  added  to  sensitize  the  student  to  the  role  of 
cultural  interactions  in  JIIM  operations.  3CI  is  an  85- 
item  self-report  assessment  tool  under  development 
by  the  Cognitive  Performance  Group.  An 
intermediate  version  of  the  inventory  was  added  to 
make  the  tutorial  more  engaging  to  students  and  to 
cause  students  to  reflect  on  their  attitudes  and  work 
style  and  how  these  elements  impact  operations. 
Students  complete  the  self-assessment  items  using  a 
Likert  scale  from  1-6,  where  1  meant  “Strongly  disagree”  and  6  meant  “Strongly  agree.” 
Feedback  on  each  of  six  scales  is  given  in  the  form  of  short  paragraphs  based  on  whether  the 
scale  score  is  above,  about  equal  to,  or  below  the  mean,  calculated  from  a  sample  of  641  military 
members  across  the  Services.  Feedback  is  general  and  statements  are  provided  for  each  of  the  six 
scale  scores  within  the  3CI.  For  example,  feedback  for  the  Willingness  to  Engage  scale  is  as 
follows  when  the  score  is  below  average:  “You  scored  somewhat  lower  than  average  on 
Willingness  to  Engage.  People  who  score  lower  may  need  to  work  harder  on  getting  to  know  the 
different  players  in  the  JIIM  environment.  This  is  an  important  skill  to  develop  as  it  will  assist 
you  in  establishing  relationships  that  allow  you  to  understand  others’  perspectives  and 
goals. ’’The  student  is  also  presented  with  the  average  scale  score  and  their  own  scale  score. 

Tutorial  version  3  -  November  2009 

For  the  third  version,  a  tutorial  called  The  Interagency  Management  System  in  Action! 
was  added  to  teach  the  IMS.  We  also  added  pre-  and  post-tests  to  the  Sudan,  Haiti,  Afghanistan, 
and  IMS  tutorials. 


Stability  Operations  in  Joint, 

Interagency,  Intergovernmental,  and 
Multinational  (JIIM)  Environments  Tutor 

Cross-Cultural  Competence 
Inventory  (3CI) 

ooo  o 


The  interagency  management  system  in  action! 

The  Interagency  Management  System  in  Action ! 
tutorial  was  designed  as  an  introduction  to  the  IMS.  It 
enabled  students  to  leam  about  the  IMS  structure, 
governance,  and  Stability  Operations  doctrine  and  to 
apply  this  knowledge  in  a  generic,  hypothetical 
situation.  This  tutorial  focused  on  the  IMS  and  when 
and  how  it  operates,  rather  than  on  the  geography  and 
politics  of  a  particular  region. 

Operations  in  JIIM  environments  will  require  new 
and  different  ways  of  thinking  and  collaborating.  This 
environment  will  require  a  shared  understanding  of 
both  problems  and  potential  solutions  offered  by  each  JIIM  partner.  This  tutorial  addressed  the 


11 


challenges  and  structures  for  collaboration,  and  it  was  designed  to  develop  critical  thinking  and 
reasoning  skills  needed  to  conduct  successful  Stability  Operations. 


Tutorial  interactions 

The  expected  benefit  of  the  tutorial  approach  is  that  problems  challenge  students  to  apply 
what  they  already  know.  The  scenarios  help  students  with  diverse  backgrounds  and  proficiency 
levels  leam  the  declarative  knowledge  more  effectively  by  providing  specific  situations  that 
cause  students  to  identify  the  important  issues  and  questions.  In  this  way,  the  scenarios  provide 
students  with  specific  questions  that  motivate  them  to  learn  the  doctrine  and  country-specific 
background  infonnation  in  an  active,  goal-directed  way.  For  example,  a  scenario  the  Security  in 
Sudan  tutorial  discusses  the  Integration  Planning  Cell  (IPC)  and  prompts  the  student  to  select 
skills  that  would  be  useful  to  include  in  the  IPC,  as  shown  in  Figure  2.  By  contrast,  traditional 
training  often  requires  students  to  passively  absorb  large  amounts  of  information  up  front  before 
applying  it  to  solve  problems. 


1 TaskGuide  Execution  Tool  -  Tutorial  2  -  Security  in  Sudan 


Tutorial  2  -  Security  in  Sudan 

Module  3  -  Integration  Planning  Cell  Staff  Skills 


Let's  discuss  the  Integration  Planning  Cell  and  how  it  supports  the  integration  of  the  civilian  and  military 
planning  efforts.  As  you  recall,  the  CRSG  specified  the  following  MMEs: 

1.  Immediate  humanitarian  needs  addressed. 

2.  Peace  and  stability  in  the  Three  Areas  (Abyei,  Blue  Nile,  and  Southern  Kordofan)  and  Darfur. 

3.  Military  transformed  and  demobilization,  disarmament,  and  reintegration  initiated,  bolstering 
confidence  of  the  Three  Areas  and  Darfur. 

4.  A  functioning  Government  of  National  Unity  (GNU)  per  Comprehensive  Peace  Agreement  that 
implements  wealth  and  power  sharing  provisions. 

5.  Economic  and  social  infrastructure  and  services  strengthened  in  targeted  communities  in  Southern 
Sudan,  including  receiving  returnees. 

6.  More  responsive  and  participatory  government  in  Southern  Sudan  in  accordance  with  the  CPA. 

Since  our  planning  team  is  focused  on  planning  for  security,  select  3  skills  and  types  of  IPC  members 
from  the  list  below  that  you  think  would  be  the  most  helpful  addition  to  our  team  that  is  focusing  on 
the  security  sector  issues  and  tasks. 


'  humanitarian  assistance  (USAID) 


conflict  resolution  (DOS) 
governance  (DOS) 


— I  justice  (DOJ) 

U  criminal  investigation  (FBI) 

—  economic  development  (Treasury) 
n  infrastructure  development  (USAID) 
1  ■!  agricultural  development  (DOA) 


© 

© 

Figure  2.  Security  in  Sudan  tutorial  prompts  for  useful  skills  to  include  in  the  Integration  Planning  Cell 


12 


Before  providing  feedback  on  the  student’s  selections,  the  tutorial  asks  the  student  to 
identify  the  key  drivers  of  conflict  as  shown  in  Figure  3.  This  screen  refers  the  student  to 
relevant  country-specific  information  (United  Nations  analysis  of  Sudan)  and  doctrine  (a  chapter 
from  FM3-07  that  discusses  Interagency  Conflict  Assessment  Framework)  accessible  via 
hyperlinks.  In  this  manner,  the  tutorial  poses  a  specific  question  that: 

•guides  the  student’s  thinking  with  respect  to  the  original  question  about  useful  IPC  skills 

•motivates  and  focuses  the  student’s  acquisition  of  knowledge  about  Sudan  and  conflict 
assessment. 


T a k G u i d e  Exe c .. :  on  c o  -  u : o  r  ■  a  1  2  -  S ec.r  t >  i n  S « c a n  _ 1  —  l—^ggl 

Tutorial  2  -  Security  in  Sudan 

Module  3  -  Integration  Planning  Cell  Staff  Skills 

1  Key  Drivers  of  Conflict 

Before  we  continue  discussing  useful  IPC  staff  skills,  let's  make  sure  we  understand  the  situation. 

What  are  some  key  drivers  of  conflict  in  this  situation? 

4  ► 

You  may  find  it  helpful  to  review  the  following  documents  in  the  Library: 

1.  ExcerDts  from  U.N.  Analvsis  of  Sudan.  Environment.  Governance  and  Securitv  (2007) 

2.  FM3-07.  ADDendix  D.  Interaaencv  Conflict  Assessment  Overview 

o 

Figure  3.  Security  in  Sudan  tutorial  poses  a  question  to  guide  the  student’s  thinking  and  to  motivate  the 
acquisition  of  country-specific  and  doctrinal  knowledge 
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When  the  tutorial  asks  the  student  a  free-text  question,  the  tutorial  does  not  evaluate  the 
student’s  answer  because  it  is  technically  difficult  for  software  to  interpret  free  text  reliably. 
However,  the  tutorial  presents  a  reference  answer  that  students  can  compare  with  their  own 
answers,  as  shown  in  Figure  4. 


Tutorial  2  -  Security  in  Sudan 

Module  3  -  Integration  Planning  Cell  Staff  Skills 
Key  Drivers  of  Conflict  -  Reference  Answer 


You  wrote  that  key  drivers  of  conflict  include  the  following: 


Compare  your  list,  above,  with  the  reference  list  below: 

1.  Severe  poverty,  displacement,  and  human  insecurity; 

2.  Access  to  and  control  over  oil-producing  areas  and  the  revenues  these  generate; 

3.  Limited  ability  of  civilian  security  and  justice  to  deal  with  violence  and  criminality; 

4.  The  continued  existence  of  armed  groups  that  have  the  potential  to  jeopardise  CPA 
implementation; 

5.  Disputes  and  competition  over  cattle,  marriage,  land,  borders,  natural  resources,  and 
other  scarce  resources,  exacerbated  by  environmental  degradation; 

6.  Inequitable  recovery,  development  and  access  to  resources; 

7.  Presence  of  large  numbers  of  small  arms  and  light  weapons  in  the  hands  of  civilians; 

8.  Manipulation  of  ethnic,  racial  and  religious  identities  for  political  purposes; 

9.  Militarisation  caused  by  prolonged  exposure  to  conflict  and  breakdown  and  marginalization 
of  traditional  conflict  management  mechanisms;  and 

10.  Conducting  and  implementing  the  elections  due  to  be  held  in  2009.  Both  Northern  and 
Southern  Sudan  have  experienced  continued  violent  conflict  and 

11.  North-South  tensions  over  contested  areas. 


& 

© 

© 

© 

© 

Figure  4.  Security  in  Sudan  tutorial  provides  a  reference  (or  expert)  answer  that  students  can  compare 
with  their  own  answers. 
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This  question  about  key  drivers  of  the  conflict  serves  as  a  hint  that  helps  the  student  reflect 
on  which  IPC  skills  are  important.  The  tutorial  then  provides  a  second  opportunity  for  the  student 
to  select  important  IPC  skills,  as  shown  in  Figure  5. 


\j>  i  TaskGuide  Execution  Tool  -  Tutorial  2  -  Security  in  Sudan  _ 


Tutorial  2  -  Security  in  Sudan 

Module  3  -  Integration  Planning  Cell  Staff  Skills 
IPC  Staff  Skills  vs.  Drivers  of  Conflict  -  Modify 


You  can  change  the  set  of  IPC  skills  that  you  feel  are  most  useful  to  the  JPG: 


humanitarian  assistance  (USAID) 
conflict  resolution  (DOS) 


m 

k'  governance  (DOS) 
justice  (DOJ) 

B  criminal  investigation  (FBI) 

I  ■  economic  development  (Treasury) 

"  infrastructure  development  (USAID) 
—  agricultural  development  (DOA) 


©©© 


© 


Figure  5.  Security  in  Sudan  tutorial  re -prompts  the  student  to  select  useful  IPC  skills. 
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Finally,  the  tutorial  provides  feedback  by  showing  the  preferred  answer  and  the  student’s 
answer,  along  with  rationale.  As  shown  in  Figure  6,  the  student’s  selected  choices  are  checked, 
and  the  preferred  choices  are  displayed  in  blue. 


-£>>  TaskGuide  Execution  Tool  -  Tutorial  2  -  Security  in  Sudan 

Tutorial  2  -  Security  in  Sudan 

Module  3  -  Integration  Planning  Cell  Staff  Skills 
IPC  Staff  Skills  Feedback 


Compare  the  IPC  staff  skills  you  checked  with  the  reference  answers  displayed  in  blue: 

®  humanitarian  assistance  (USAID) 

®  conflict  resolution  (DOS) 

O  governance  (DOS) 
d  justice  (DOJ) 

S  criminal  investigation  (FBI) 

C  economic  development  (Treasury) 

~  infrastructure  development  (USAID) 

LJ  agricultural  development  (DOA) 


Rationale  for  the  reference  answer: 

Our  team  has  been  tasked  to  address  the  Security  Sector.  Justice,  conflict  resolution,  and 
criminal  investigation  most  closely  align  with  Security  Sector  tasks. 

Conflict  resolution  skills  could  be  helpful  for  addressing  the  underlying  drivers  of  conflict. 

Providing  humanitarian  assistance  does  impact  all  sectors  because  it  provides  for  the  most 
basic  human  needs.  It  is  very  important  to  the  success  of  stability  operations,  however,  it  is 
not  as  closely  aligned  with  the  Stability  Sector  that  our  planning  group  has  been  tasked  to 
address. 

Other  planning  teams  have  been  created  and  tasked  to  address  the  other  sectors:  Justice 
and  Reconciliation,  Humanitarian  Assistance  and  Social  Well-Being,  Governance  and 
Participation,  and  Economic  Stabilization  and  Infrastructure. 


& 


o 


Figure  6.  Security  in  Sudan  tutorial  provides  feedback  and  rationale  for  a  multiple  choice  question. 


Some  of  the  tutorials  provided  response-specific  hints  after  the  student  has  answered  a 
multiple  choice  question.  For  example,  the  tutorial  might  point  out  relevant  information  for  each 
missing  or  incorrect  choice  selected  by  the  student.  After  presenting  the  hint,  the  tutorial 
provides  the  student  with  an  opportunity  to  change  their  answer  to  the  original  question.  Finally, 
the  tutorial  provides  feedback  by  displaying  the  student’s  selections,  the  preferred  selections, 
rationale  for  the  preferred  selections,  and  reasons  why  incorrect  choices  selected  by  the  student, 
if  any,  are  not  preferred. 
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Knowledge  checks 

To  ensure  understanding,  examples  and  numerous  items  were  used  as  “knowledge 
checks”  before  the  more  complex  tutorials  were  accessed.  The  themes  were  also  introduced  as 
skills  to  be  developed  for  success  in  the  JIIM  environment  to  sensitize  the  learner  to  the  fact  that 
the  elements  of  success  were  more  complex  than  understanding  doctrine  and  processes.  The 
tutorial  presented  a  series  of  situations  and  prompted  students  to  identify  relevant  themes.  An 
example  situation  and  prompt  is  shown  in  Figure  7. 


I  '  i— -I 
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Introduction  to  Operations  in  JIIM  Environments 

Module  2  -  High-Level  Skills  in  JIIM  Environments 
Check  all  the  skills  that  apply  in  SITUATION  ONE. 


State  Department  official  participating  in  counter-narcotics  operations.  What  I  learned  in  Mexico 
more  than  anything  else  was  just  what  a  bad  image  the  State  Department  has  to  a  lot  of  other  agencies. 
Following  our  being  in  the  middle  of  a  big  shoot-out,  I  was  surprised  at  how  many  meetings  I  would  go  to 
where  they  would  say,  "thank  you,  thank  you,  thank  you  for  taking  such  an  active  role  in  this."  And  I  got 
the  feeling,  that  other  agencies  genuinely  believe  that  State  Department  officials  tend  to  stay  behind  their 
desks,  and  they  tend  to  be  such  risk  avoiders  and  controversy  avoiders  that  they’re  really  kind  of  useless 
when  you  get  into  a  situation  where  you  have  to  make  a  decision,  and  have  to  get  something  done, 


□  1.  Understanding  the  context 

□  2.  Understanding  other  participants 
U3.  Shifting  perspective 

11  4.  Establishing  common  ground 

□  5.  Building  capacity  in  the  team 

□  6.  Visualizing  the  outcomes 

□  7.  Supporting  information  exchange 

□  8.  Maintaining  flexibility 


0  o  o 


Figure  7.  Example  situations  prompt  students  to  identify  relevant  themes 

Situation  judgment  tests 

Situation  Judgment  Tests  (SJTs)  were  added  to  the  Sudan  tutorial  and  were  labeled  as 
Practical  Exercises.  These  practical  exercises  used  a  rich  context  and  allowed  the  students  to 
make  complex  situation  assessments  without  entering  free-form  texts,  because  SJTs  use  a 
multiple  choice  fonnat.  SJTs  are  aimed  at  assessing  knowledge  of  concepts,  integration  of 
multiple  concepts  and  application  of  that  knowledge  to  realistic  dilemmas. 

To  develop  an  SJT,  the  author  must  understand  typical  dilemmas  and  issues  faced  in  the 
environment  and  domain  of  practice.  Drawing  on  prior  understanding  of  conflicts  and 
assessments  at  the  tactical  and  operational  level  of  Stability  Operations,  rich  descriptions  were 
developed  of  situations  and  a  statement  of  the  best  response  possible  that  took  into  account  all 
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the  variables  outlined  in  the  background  description  and  potential  second  and  third  order 
consequences.  Then  up  to  three  other  statements  that  describe  plausible  responses  to  the  situation 
were  developed.  For  each  statement,  feedback  was  developed  on  what  was  good  about  the 
response  and  what  could  cause  potential  problems  if  the  response  was  selected  as  the  desired 
action.  Responses  to  SJT  items  can  also  be  used  to  sum  up  performance  to  a  user  at  the  end  of  a 
module  of  tutorial.  Summing  up  the  level  of  performance  could  provide  more  diagnostic 
feedback.  It  can  also  be  part  of  an  ongoing,  continual  assessment  to  see  if  the  user  grasps 
concepts  and  integrates  concepts  more  effectively  as  the  training  progresses.  To  provide  the  most 
diagnostic  feedback,  the  developer  must  identify  domain  knowledge  and  concepts  or  themes  for 
the  domain  to  each  SJT  and  each  response  during  the  authoring  process.  Feedback  can  then 
indicate  how  successful  the  choices  were  across  SJTs  and  link  the  choices  to  how  well  particular 
knowledge  was  applied.  Figure  8  through  Figure  1 1  show  the  background  information,  prompt, 
and  feedback  for  an  example  SJT. 
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Practical  Exercise  1:  Crowd  Control  and  Maintaining  Positive  Relations 


Please  read  the  following  incident  carefully.  Following  this  background  information,  you  be  given  a  specific 
situation  and  asked  to  choose  the  best  course  of  action  (COA),  taking  into  account  the  following  information. 


Background 


You  have  been  recently  deployed  to  a  country  in  Northeast  Africa.  You  are  assigned  as  the  leader  of  a  multi-national 
reconstruction  team  whose  mission  is  to  promote  stability  and  security  to  a  particular  war  torn  district.  To  date,  you  have 
been  operating  in  this  area  for  a  few  months.  While  you  have  not  personally  experienced  insurgent  activity,  you  are  aware  it 
does  occur  on  occasion.  Additionally,  locals'  feelings  towards  Americans  are  mixed. 

As  your  convoy  approaches  the  outskirts  of  a  village  with  unknown  sympathies,  a  member  of  your  convoy  accidentally 
discharges  his  weapon  into  the  hillside.  Though  no  one  is  injured,  you  notice  a  large  crowd  of  locals  gathering  around  you 
and  your  Soldiers.  The  crowd  is  growing  noticeably  anxious,  and  many  individuals  are  shouting  and  pointing.  At  this  stage, 
calming  the  crowd  is  top  priority  in  order  to  maintain  security,  and  your  interpreter  thinks  he  overheard  someone  mention 
getting  their  guns.  You  attempt  to  perceive  the  situation  from  the  perspective  of  the  villagers  and  their  leader  in  order  to 
initiate  your  next  action. 


Figure  8.  Example  SJT  -  background  page  1 
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Introduction  to  Operations  in  JIIM  Environments 

Module  3  Exercise  1  -  Crowd  Control  and  Maintaining  Positive  Relations 


Background  Page  2 


You  observe  that  the  crowd  currently  has  no  weapons,  but  their  fists  are  clenched  and  the  there  is  loud  yelling.  You  survey  the 
crowd  and  note  that  there  are  several  children  but  no  women  close  to  the  road  where  the  convoy  has  stopped.  None  of  the 
civilians  is  on  or  near  any  of  the  vehicles,  Based  on  your  knowledge  of  the  region,  and  the  composition  and  behavior  of  the 
crowd  you  assess  that  the  crowd  is  made  up  of  curious,  energetic  onlookers  who  are  demonstrating  non-threatening 
behavior.  You  look  around  and  see  stress  on  the  face  of  your  Soldiers,  You  don't  want  to  escalate  the  situation  into  a 
confrontation. 


Figure  9  .  Example  SJT  -  background  page  2 


Introduction  to  Operations  in  JIIM  Environments 

Module  3  Exercise  1  -  Crowd  Control  and  Maintaining  Positive  Relations 

What  is  the  Best  COA? 


Given  this  situation,  and  what  you  believe  to  be  the  crowd's  intentions,  what  is  your  best  course  of  action  (COA)  at 
this  point  to  prevent  a  confrontation  and  control  the  crowd? 

For  a  quick  reference  on  planning,  you  may  want  to  review  this  chart  found  within  military  doctrine  on  Post-Conflict 
Reconstruction.  Essential  Tasks: 

□  1.  While  still  at  the  outskirts  of  the  village,  I  would  stop  my  troops  and  deploy  the  convoy  in  a  herringbone  formation  with 
weapons  muzzle  orientation  pointed  down  to  the  ground,  My  interpreter  and  I  would  approach  the  crowd  and  ask  to  speak  to 
the  village  elders  or  leaders. 

□  2.  I  would  halt  the  convoy  perform  the  "React  to  Human  Obstacles"  drill:  identify  the  crowd's  activity  and  intentions  as  far 
away  as  possible;  bypass  if  possible;  assume  hostile  intent  on  the  part  of  the  crowd;  adopt  aggressive  posture;  position 
CREW-equipped  gun  truck  in  overwatch  position;  push  vehicles  slowly  through  crowd  while  sounding  horns;  use  Escalation  of 
Force  (EOF)  measures  if  the  crowd  refuses  to  move;  forward  Situation  Report  (SITREP)  to  higher  headquarters;  and  continue 
the  mission.  When/if  confronted  by  a  crowd:  do  not  stop;  always  assume  hostile  intent;  do  not  appease,  feed,  or  give  them 
anything;  and  force  your  way  through. 

D  3.  I  would  lead  the  convoy  slowly  into  the  village,  have  Soldiers  dismount  and  point  their  weapons  to  the  ground  and  kneel 
down  on  one  knee.  My  interpreter  and  I  would  approach  the  crowd  and  ask  to  speak  to  the  village's  leadership. 

D  4.  I  would  determine  who  is  in  charge  of  this  village  in  order  to  clarify  what  occurred  at  the  outskirts  of  their  village  and 
begin  discussions  addressing  the  purpose  of  your  visit  and  identifying  the  needs  of  the  village  and  build  relationships. 


© 
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Figure  10.  Example  SJT  prompts  the  student  for  the  best  course  of  action. 
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Feedback 


Introduction  to  Operations  in  JIIM  Environments 

Module  3  Exercise  1  -  Crowd  Control  and  Maintaining  Positive  Relations 


Great  job,  this  is  an  excellent  option,  and  the  best  response.  This  COA  has  the  lowest  probability  of  provoking  the  crowd 
into  an  unmanageable  confrontation  and  helps  create  or  maintain  a  positive  relationship  with  the  local  population.  You  have 
used  cues  about  the  composition  and  behavior  of  the  crowd  (children  present;  no  weapons;  no  aggressive  approach)  and 
your  knowledge  of  the  culture  to  understand  their  intention.  By  taking  the  perspective  of  the  civilians,  you  have  sized  up  the 
situation  including  the  fact  that  there  is  a  low  potential  threat.  You  understood  that  assuming  a  less  aggressive  posture,  with 
muzzles  down  and  soldiers  on  one  knee,  you  have  sent  a  message  to  the  crowd  that  there  was  no  hostile  intent.  It  makes 
sense  to  use  your  interpreter  to  communicate  with  the  group,  whether  or  not  elders  or  leaders  are  present  as  they  may 
not  be  immediately  avialable.  Furthermore,  this  COA  shows  an  understanding  of  the  local  culture  -  that  individuals  in  this 
area  acknowledge  and  accept  that  decision  making  power  is  in  the  hands  of  local  elders  and  controlling  a  disturbed  crowd 
must  be  done  through  this  individual,  as  well  as  the  tendency  for  people  here  to  form  crowds  and  to  be  loud  with  no  hostile 
intent. 


o 
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Figure  11  Example  SJT -feedback 


FINDINGS  AND  DISCUSSION 

Training  Effectiveness  Evaluation 

The  goal  of  the  evaluation  was  to  identify  the  effectiveness  of  the  Interagency  Management 
System  in  Action!  tutorial  by  analyzing  the  differences  between  the  pre-  and  post-test  scores. 

Data  was  examined  at  the  .05  level  of  statistical  significance. 

Pre-  and  post-test  results 

The  average  number  correct  on  the  multiple  choice  pre-test  was  11.6  out  of  25  questions 
(46.4%),  and  the  average  number  correct  on  the  post-test  was  17.8  of  25  (71.2%),  a  difference 
that  was  significant  using  an  independent  samples  t-test  (p<0.05). 

An  item  analysis  was  perfonned  to  determine  which  of  the  multiple  choice  test  items  best 
discriminated  the  performance  within  the  group.  Student  scores  for  1 1  of  the  25  items  showed 
significant  difference  between  the  pre-  and  post-tests  (p<.05).  Appendix  A  contains  the  full 
knowledge  test  with  correct  answers  indicated  and  values  for  each  item  indicating  those  that 
showed  a  significant  difference  in  discriminating  between  the  pre-  and  post-tests. 

Reaction  survey  results 

The  Reaction  Survey  responses  were  analyzed  by  looking  at  the  mean  score  on  each  of 
the  questionnaire  items.  The  scoring  for  all  items  was  based  on  a  five-point  scale: 

1  Strongly  Disagree 

2  Disagree 

3  Neutral 

4  Agree 

5  Strongly  Agree 
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The  Reaction  Survey  was  separated  into  three  categories.  The  first  category  focused  on 
the  participants’  overall  attitudes  toward  the  training.  The  most  positive  results  were  participants 
agreeing  that: 

•The  Interagency  Management  System  in  Action!  tutorial  enhanced  my  understanding  of  the 
concepts  of  the  course  (mean=4.05) 

•The  tutorial  helped  me  to  look  at  Interagency  operations  from  several  perspectives 
(mean=4.00) 

•The  technology  was  easy  to  download  and  use  (mean=3.95) 

The  most  negative  result  was: 

•The  tutorial  took  too  long  to  complete  (mean=3.45) 

The  results  of  this  section  of  the  survey  are  displayed  in  Table  2. 


Table  2,  Reaction  Survey  Items  and  Means  for  Overall  Attitude  Toward  the  Training. 


Reaction  Survey  Question 

Mean 

1 .  The  “IMS  in  Action”  tutorial  enhanced  my  understanding  of  the  concepts  of  the 
course. 

4.05 

2.  The  tutorial  was  often  boring. 

2.55 

3.  The  tutorial  was  a  valuable  learning  experience  for  me. 

3.90 

4.  The  tutorial  was  easy  to  use. 

3.55 

5. 1  would  like  to  participate  in  more  tutorials  like  this  in  the  future. 

3.55 

6.  The  tutorial  helped  me  to  look  at  interagency  operations  from  several 
perspectives. 

4.00 

7.  The  material  presented  in  the  tutorial  was  clear  and  easy  to  understand. 

3.60 

8.  The  tutorial  took  too  long  to  complete. 

3.45 

9.  The  technology  was  easy  to  download  and  use. 

3.95 

10.  Using  the  tutorial  has  improved  my  ability  to  understand  other  organizations 
and  their  functions. 

3.85 

1 1 . 1  am  better  able  to  understand  the  desired  role  of  the  US  military’s  role  in 

Stability  Operations. 

3.70 

12.1  feel  better  able  to  implement  a  Whole  of  Government  approach  to  Stability 
Operations. 

3.70 
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The  second  part  of  the  Reaction  Survey  focused  on  how  well  the  tutorial  supported 
various  learning  objectives.  The  scoring  for  each  learning  objective  was  based  on  a  five-point 
scale  where  a  “1”  signified  that  they  did  not  feel  that  the  training  supported  the  learning  objective 
and  a  “5”  illustrated  strong  agreement  that  it  supported  the  learning  objective.  Participant 
reactions  toward  the  tutorial  teaching  the  learning  objectives  ranged  between  3.35  and  3.95. 

The  most  positive  results  were  agreements  by  participants  that  the  tutorial  supported  the 
following  learning  objectives: 

•Identify  the  wide  array  of  actors  within  Stability  Operations  in  the  U.S.  Government  and 
larger  Stability  Operations  community  (mean  =  3.95), 

•Explain  the  Interagency  Management  System  as  a  method  for  facilitating  the  WGA 
approach  (mean=3.95),  and 

•Explain  the  U.S.  Whole  of  Government  Approach  (WGA)  to  Stability  Operations 
(mean=3.90). 

Table  3  shows  the  items  and  the  response  means  for  this  section  of  the  survey. 

Table  3.  Reaction  Survey  Mean  Ratings  for  How  Well  the  IMS  Tutorial  Supported  Learning  Objectives. 


Learning  Objective 

Mean 

13.  Identify  the  wide  array  of  actors  within  Stability  Operations  in  the  U.S.  Government 
and  larger  Stability  Operations  community. 

3.95 

14.  Explain  the  role  of  the  Military  in  accordance  with  the  WGA. 

3.55 

15.  Explain  the  Interagency  Management  System  as  a  method  for  facilitating  the  WGA 
approach. 

3.95 

16.  Explain  the  U.S.  Whole  of  Government  Approach  (WGA)  to  Stability  Operations. 

3.90 

17.  Describe  the  IPC  portion  of  the  Interagency  Management  System  as  a  method  for 
facilitating  the  WGA. 

3.80 

18.  Differentiate  the  capabilities  of  the  different  Interagency  organizations  in  Stability 
Operations. 

3.60 

19.  Describe  how  the  WGA  and  DoD  organizations  align  both  structure  and  processes 
when  the  IMS  is  initiated. 

3.70 

20.  Differentiate  the  capabilities  of  the  different  Interagency  organizations  in  Stability 
Operations. 

3.70 

2 1 .  Describe  the  Interagency  Management  System  (ACT  specifically)  as  a  method  for 
facilitating  the  WGA  at  the  JTF  level. 

3.70 

22.  Define  how  the  U.S.  military  can  support  goals  of  other  U.S.  government  agencies  in 
Stability  Operations. 

3.70 

23.  Describe  how  the  WGA  and  DoD  organizations  align  both  structure  and  processes 
when  the  IMS  is  initiated. 

3.70 

24.  Compare  different  methods  of  solving  the  same  problem. 

3.35 
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Table  3.  (continued) 


25.  Differentiate  the  capabilities  of  the  different  players  in  Stability  Operations. 

3.55 

26.  Demonstrate  an  understanding  of  the  perspectives  of  others  around  a  single  situation 
or  decision. 

3.45 

27.  Identify  common  goals  among  players  in  Stability  Operations. 

3.60 

28.  Identify  typical  sources  of  friction  among  different  players  in  Stability  Operations. 

3.35 

The  third  section  containing  written  feedback  from  participants  showed  that  the  tutorial 
was  a  useful  learning  experience  and  that  it  could  be  improved  in  some  areas,  as  shown  below  in 
Table  4. 

Table  4.  Reaction  Sun’ey  Comments  about  Strengths  and  Weaknesses  of  the  Tutorial 


Positive  Feedback  Places  for  Improvement 


•Good  in-depth  material  and  introduction  •Material  too  dense/difficult/too  time  consuming 
to  IMS  *Too  many  abbreviations/acronyms  not  spelled  out 

•Forces  student  to  be  able  to  apply  ^Software  awkward/Not  user  friendly  (e.g.,  No 

material  ability  to  move  around  within  tutorial  or  go  back 

without  losing  data). 

•Difficulty  getting  the  software  to  install 
•Add  video/ audio/ diagrams/pictures/interactive 
component. 


Limitations  of  the  findings 

The  participants  in  this  evaluation  were  motivated  to  participate  in  this  tutorial  as 
indicated  by  the  Reaction  Survey  data.  Many  researchers  have  identified  the  importance  of 
student  motivation  to  leam  as  a  key  factor  in  any  effective  training  program  (Baxter,  2003;  Ilgen 
&  Klein,  1989;  Locke  &  Latham,  1984;  Luthans  &  Stajkovic,  1999).  The  motivational 
characteristic  of  this  type  of  tutorial  is  clearly  a  factor  in  its  favor  compared  to  other  training 
media.  A  number  of  reasons  have  been  identified  which  help  digital-based  training  systems 
achieve  their  motivating  effects.  These  include  clear  cut  goals,  fast  pace,  immediate  feedback, 
and  variable  or  increasing  levels  of  challenge  (Chaffin,  Maxwell,  &  Thompson,  1982). 

Although  a  test-retest  without  the  tutorial  training  would  have  acted  as  a  control  group  to 
measure  any  gains  in  post-test  scores  simply  as  a  function  of  practice  taking  the  test,  there  were 
not  enough  students  available  for  this  activity. 

Participant  acceptance  of  training  is  an  important  criterion  and  one  that  is  heavily  relied  upon  in 
training  evaluation  research.  A  caution,  however,  is  the  increasingly  well-documented  finding 
that  training  interventions  that  produce  difficulties  for  trainees  during  acquisition,  and  that  may 
frustrate  trainees,  may  actually  produce  better  learning  as  evidenced  by  transfer  and  retention. 
Baddeley  and  Longman  (1978),  for  example,  found  that  training  conditions  most  preferred  by 
students  resulted  in  less  learning.  A  conclusion  from  Schmidt  and  Bjork’s  (1992)  review  of  this 
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literature  is  that  valuable  training  interventions  may  go  undetected  because  rarely  do  evaluations 
assess  transfer  and  retention.  Thus,  a  caution  is  that  the  present  evaluation  results  apply  to 
baseline  knowledge  acquisition  only  and  are  not  indicative  of  actual  transfer  to  field  or  exercise 
perfonnance  which  was  not  assessed  in  this  project. 

Authoring  tool  usability  assessment 

A  fonner  instructor  at  Joint  Forces  Staff  College  reviewed  the  Task  Guide 
documentation,  received  a  demonstration  of  the  tool,  and  made  several  practical  changes  to  the 
tutorial.  Based  on  this  experience,  he  concluded  that  the  tool  was  useful  for  creating  effective 
instructional  content  and  that  one  hour  of  preparatory  document  review  and  two  and  a  half  hours 
of  instruction  were  sufficient  to  enable  him  to  make  simple  changes  to  an  existing  tutorial.  He 
estimated  that  three  additional  hours  of  document  review  and  instruction  would  enable  him  to 
make  major  changes;  including  creating  a  tutorial  from  scratch,  assuming  that  the  instructional 
design  of  the  tutorials  had  already  been  carried  out.  Pre-requisites  for  successful  use  of  the  tool 
include:  attention  to  detail,  advanced  preparation  of  changes,  detailed  knowledge  of  the  subject 
matter  and  instructional  plan,  documentation  of  changes  and  version  control,  and  a  basic  level  of 
computer  familiarity  common  among  instructors  who  utilize  education  technology 
systems/networks.  His  response  to  the  questionnaire  is  presented  in  Appendix  C. 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  collection  of  training  tutorials  is  a  training  product  which  provides  a  baseline 
understanding  for  a  potentially  large  and  diverse  training  audience  for  an  area  of  practice  that  is 
vital  to  operations  but  not  yet  widely  integrated  in  our  education  and  training  systems.  Few 
opportunities  exist  to  share  this  infonnation  among  the  variety  of  participants  who  must  engage 
in  these  operations.  The  doctrine  and  knowledge  being  produced  to  support  Stability  Operations, 
the  IMS,  and  non-kinetic  operations  is,  in  general,  scattered  and  diverse.  The  amount  and  type  of 
reference  material  already  in  the  tool  provides  a  useful  reference  library  and  place  for  different 
users  or  user  groups  to  potentially  store  additional  material  as  they  use  the  product.  This  tool  fills 
a  need  that  no  other  distance  learning  product  is  currently  addressing,  and  it  does  so  in  a  manner 
that  supports  the  range  of  learning  from  knowledge  acquisition  of  basic  concepts  to  the 
integration  and  application  of  those  concepts.  In  addition,  the  tool  provides  relevant  processes 
and  procedures,  as  well  as  insight  into  the  high-level  cognitive  skills  needed  for  successful 
application  in  an  operational  environment  where  collaboration  is  at  its  most  challenging.  The 
specification  of  learning  objectives  for  each  module  within  each  tutorial  supports  the 
instructional  planner  in  integrating  the  product  into  a  larger  training  program.  The  authoring  tool 
that  is  available  with  the  product  also  allows  for  scenarios  and  assessment  items  to  be  modified 
or  added  as  learning  objectives  change  and  new  examples  are  generated  from  practice. 

Transition  to  Operational  Use 

The  tutorials  have  been  selected  for  use  by  the  Joint,  Interagency,  and  Multinational 
Planner’s  Course  (JIMPC),  offered  at  Joint  Forces  Staff  College.  The  IMS  Tutorial  has  been 
reviewed  by  members  of  the  Office  of  the  Secretary  of  Defense  (OSD)  Personnel  and  Readiness 
and  the  Department  of  State.  It  is  under  consideration  by  US  JFCOM  to  educate  Combatant 
Commander  and  Joint  Task  Force  staffs  on  the  Interagency  Management  System  who  cannot 
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attend  the  S/CRS  Foundations  course  and  must  rely  on  distance  learning  methods.  The  tutorials 
are  also  being  evaluated  for  wider  use  by  other  schools  at  Joint  Forces  Staff  College. 

Recommendations 

In  general,  digital  scenario-based  training  is  most  effective  when  it  is  embedded  in  a 
larger  training  program  containing  a  number  of  other  training  elements  such  as  classroom  and 
live  training  (Prensky,  2001).  Integration  should  include: 

•  Identification  of  specific  course  training  objectives  that  can  be  met  with  the  tutorial. 

•  Development  of  scenarios  that  allow  deliberate  practice  with  respect  to  the  learning 
objectives. 

•  Development  of  performance  assessment  tools  so  that  achievement  of  learning  objectives 
can  be  measured. 

•  After  action  review  to  provide  feedback  to  trainees  regarding  how  well  learning 
objectives  were  achieved. 

The  feedback  provided  in  the  tutorials  presents  ideas  for  further  performance  assessment 
and  after  action  review  in  a  more  comprehensive  training  program.  Feedback  provided  by 
additional  user  groups  can  be  incorporated  to  improve  the  tutorials,  and  the  results  of  different 
uses  of  the  tutorials,  such  as  standalone  use  or  integration  into  a  larger  curriculum,  should  be 
documented. 

The  addition  of  more  SJTs  can  serve  as  learning  tools  to  practice  using  concepts  but  can 
also  be  used  as  an  ongoing  assessment.  Summing  up  findings  across  SJTs  can  also  provide  an 
after-action  review  (AAR)  capability.  The  SJTs  in  the  current  tool  can  serve  as  examples  for 
instructors  who  want  to  leam  to  develop  these  short  vignette-based  assessments  using  current 
experiences  and  historical  cases.  More  sophisticated  Socratic  dialog  techniques  could  be  added 
to  these  tutorials  and  to  new  tutorials  that  take  greater  advantage  of  Task  Guide’s  ability  to 
present  infonnation,  provide  hints,  and  pose  questions  adaptively  to  guide  the  student’s  thought 
processes. 

In  conclusion,  to  successfully  use  the  product,  adopters  must  integrate  it  with  their 
objectives  and  detennine  how  they  want  to  use  the  tool  for  best  effect.  The  entire  tool  can  be 
used  to  establish  a  baseline  for  a  heterogeneous  training  audience  to  enhance  later  engagement  in 
live  exercises.  The  tool  can  also  be  used  as  part  of  a  larger  curriculum.  Subsets  of  the  tool  can  be 
used  to  introduce  concepts.  For  example,  adopters  could  use  only  the  Interagency  Management 
System  in  Action!  tutorial  as  an  introduction  to  the  IMS  to  meet  some  objectives  of  a  larger 
curriculum. 
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APPENDIX  A 

IMS  TUTORIAL  KNOWLEDGE  TEST  ITEMS  AND  SIGNIFICANCE 
VALUES  FOR  DISCRIMINATING  PRE-  AND  POST-TEST  SCORES 


A-l 


1.  The  purpose  of  an  integration  planning  cell  is  a 

a. To  support  civil-military  communication  and  harmonize 

civilian  military  planning 

b. To  create  a  USG  R  and  S  Operations  Plan 

c. To  create  a  USG  R  and  S  Implementation  Plan 

d. All  of  the  above 

e. None  of  the  above 

0.002* 

2.  On  an  IPC,  the  State  Department  works  on  all  of  the  following  d 
except: 

a. Planning,  Operations,  and  Management 

b. Criminal  Justice  and  Policing 

c. Diplomacy  and  Governance 

d. Essential  Services 

e. Diplomatic  Security 

0.016* 

3.  Criminal  Justice  and  policing  is  handled  by  e 

a.  State  Department 
b  Justice  Department 

c.  Homeland  Security 

d. USAID 

e. All  of  the  Above 

0.124 

4.  The  Interagency  Planning  Framework  used  by  the  US  d 

Government  consists  of 

a. Storming,  Fonning,  Norming,  Performing 

b. Assessment,  Execution,  Evaluation 

c.  Assessment,  Strategy  Development,  Execution 

d. Policy  Formation,  Strategy  Development,  Implementation 

Planning 

e. None  of  the  above 

0.029* 

5.  The  elements  of  a  plan  that  are  necessary  to  achieve  the  b 

Mission  Conflict  Transformation  Goal  are  called: 

a. Overarching  Policy  Goals 

b. Major  Mission  Elements 

c. Essential  Tasks 

d. Planning  Template 

e.  Implementation  Planning 

0.000* 

6.  An  example  of  a  Major  Mission  Element  is:  e 

a. Immediate  humanitarian  needs  addressed 

b. Hydrocarbons 

c.  Security  Assistance 

0.003* 

A-2 


d. Regional  Disputes 

e. All  of  the  Above 


7.  What  is  the  ACT  relationship  between  the  US  Embassy  and  the  c  0.171 

JTF 

a. ACT  works  with  the  JTF,  but  not  with  the  US  Embassy 

b. ACT  works  with  the  US  Embassy,  but  not  with  the  JTF 

c. ACT  splits  into  2  sections:  one  to  work  with  the  US  Embassy 

and  one  to  work  with  the  JTF 

d. ACT  splits  into  3  sections:  One  to  work  with  each  the  US 

Embassy,  JTF,  and  Host  Nation 

e. None  of  the  above 


8,  Which  of  the  following  is  false  regarding  the  ACT  relationship  a  0.418 

to  the  US  Embassy  and  the  JTF? 

a. ACT  participating  in  the  IMS  will  coordinate  with  many 

different  actors  and  entities  -  some  of  whom  may  not  be 
operating  under  the  system. 

b. These  other  elements  may  include  the  host  country  (public 

and  private  sector)  and  its  military,  coalition  militaries,  and 
NGOs. 

c. Within  the  IMS,  the  Joint  Force  Commander  will  have 

reoccurring  coordination  and  interaction  with  members 
representing  the  IPC,  ACT,  and  if  deployed,  the  FACT. 

d. Coordination  with  the  CRSG  will  likely  occur  through 

guidance  and  direction  received  from  OSD,  Joint  Staff,  and 
the  GCC  using  existing  procedures 

e. It  will  be  important  for  the  JTF  Commander  and  staff  to  know 

and  understand  the  mission  and  purpose  of  each  ACT 
member  agency  so  coordination  requirements  can  be  better 
supported. 


9.  As  you  transition  to  stability  and  reconstruction  activities  who  b  0. 169 

is  responsible  for  the  execution 

a.JTF  Commander 

b  Joint  and  Interagency  Responsibility 

c. IPC 

d. ACT 

e. None  of  the  Above 


lO.The  US  government  has  the  developed  an  interagency  planning  c  0.554 

and  coordination  framework  for  Stability  Operations  called: 

a. Comprehensive  approach 

b.  Joint  Forces  Command 

c.  Interagency  Management  System 

d. Civilian  Response  Forces 

e. None  of  the  above 
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11. Smart  power  refers  to: 

a. Cyber  operations 

b. A  means  to  integrate  Joint  Interagency  and  Multinational 

capabilities 

c. A  means  by  which  the  US  State  Depart  leverages  diplomatic  power 

d. The  lull  range  of  diplomatic,  economic,  military,  political,  legal 

and  cultural  tools  available  to  the  US  government 

e. A  National  Security  Objective  to  bring  coalition  partners  together  to 

conduct  Stability  Operations. 

d 

0.124 

12.  Which  of  the  following  is  likely  to  trigger  the  activation  of  the  IMS 

a. Little  to  no  US  military  involvement 

b. Low  mortality  rates  in  a  conflict 

c. Large  scale  displacement  of  people 

d. No  threat  to  US  civilians 

e. Decrease  of  USG  funded  civilian  programs 

c 

0.058 

13.According  to  IMS  triggering  guidance  in  the  case  of  eminent  crisis 
with  Reconstruction  and  stability  or  conflict  implications,  what  senior 
officials  can  trigger  whole  of  government  planning? 

a. The  Coordinator  for  Reconstruction  and  Stability 

b. The  Secretary  of  State 

c. The  Secretary  of  Defense 

d. A  &  B 

e. B  &  C 

f. All  of  the  above 

e 

0.291 

14.The  IMS  is  intended  to  support 

a. Integrated  planning  processes 

b.  Joint  interagency  field  deployments 

c.  Joint  civil-military  operations 

d.  Shared  communications  and  information  management 

e. All  of  the  above 

f. A  &  D 

e 

0.476 

15.The  IMS  consist  of  three  functional  levels: 

a. The  Country  Reconstruction  &  Stabilization  Group  (CRSG),  The 

Integration  Planning  Cell  (IPC),  The  Advanced  Civilian  Team 
(ACT) 

b. The  Department  of  State,  The  Country  Reconstruction  & 

Stabilization  Group  (CRSG),  Advanced  Civilian  Teams  (ACT) 

c. The  Advanced  Civilian  Team  (ACT),  Field  Advanced  Civilian 

Teams  (FACT),  The  Integration  Planning  Cell  (IPC) 

d. The  Secretary  of  State,  The  Secretary  of  Defense  and  the  National 

Security  Council 

e. The  Department  of  State  Coordinator  for  Reconstruction  and 

Stability,  The  US  Ambassador  in  the  affected  country,  the 

Country  Team. 

a 

0.136 

16.The  IMS  is  not  likely  to  be  used  when 

a. Highly  complex  crisis  or  operations  which  are  national  or  securities 

priorities 

b. There  is  widespread  instability 

c. There  is  a  humanitarian  crises  that  are  handled  through  the  current 

Washington  DC  and  US  Embassy  based  system  *** 

d. A  situation  is  likely  to  require  military  operations 

e. A  foreign  state  fails  or  falls  into  crisis 

c 

0.003* 
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17.The  Integration  Planning  Cell  (IPC)  deploys  to  and  supports 

a. The  Department  of  State 

b. The  Country  Reconstruction  &  Stabilization  Group  (CRSG) 

c. The  US  Embassy  in  the  affected  country 

d. The  Joint  Task  Force 

e. The  Geographic  Combatant  Command 

e 

0.003* 

18.The  Advanced  Civilian  Team  (ACT)  deploys  to  and  supports 

a. The  Department  of  State 

b. The  Country  Reconstruction  &  Stabilization  Group  (CRSG) 

c. The  Geographic  Combatant  Command 

d. The  Joint  Task  Force 

d 

0.008* 

19.The  Field  Advanced  Civilian  Teams  (FACT)  deploys  to  and  supports 

a. The  Department  of  State 

b. The  Country  Reconstruction  &  Stabilization  Group  (CRSG) 

c. The  US  Embassy  in  the  affected  country 

d. The  Joint  Task  Force 

e. The  Geographic  Combatant  Command 

d 

0.082 

20.The  Field  Advanced  Civilian  Teams  (FACT)will  locate  In  the 

provinces  outlying  area  and  operate  under  the  direct  guidance  of  the 
Integration  Planning  Cell  (IPC) 

a. True 

b. False 

b 

0.735 

21.  The  military  representation  on  the  Country  Reconstruction  & 
Stabilization  Group  (CRSG)  comes  from 

i. The  Department  of  Defense 

ii. The  US  Embassy  in  the  affected  country 

iii. The  Joint  Task  Force 

iv. The  Geographic  Combatant  Command 

v. Joint  Staff 

a. i  &  v 

b. i  &  ii 

c. iii&  iv  &  v 

d. i  &  iv  &  v 

e. all  of  the  above 

d 

0.011* 

22.  Which  one  of  the  following  is  not  usually  part  of  the  IMS 

a.  Department  of  Justice 

b. Department  of  Defense 

c.  Department  of  Treasury 

d. Department  of  the  Interior 

e. Department  of  Education 

f. Corps  of  Engineers 

d 

0.728 

23.What  agency  is  responsible  for  diplomatic  Security 

a.  Department  of  Justice 

b. Department  of  Defense 
e.Department  of  Treasury 

d. Department  of  the  Interior 

e. Department  of  Homeland  Security 

f. Department  of  State 

f 

0.009* 
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24.The  Interagency  Conflict  Assessment  Framework  (ICAF)  is:  d  0.700 

a. The  Army  version  of  interagency  planning 

b. A  theoretical  framework  that  assigns  conflict  resolution  values  to 

countries  in  crisis 

c. Used  to  prevent  violent  conflicts 

d. A  tool  to  identify  causes  of  instability  and  to  develop  activities  to 

diminish  and  mitigate  them 

e. A  tactical  planning  process  used  by  the  other  government  agencies 

to  interpret  military  campaign  planning  into  civilian  terms 


25.The  Integration  Planning  Cell  (IPC)  leader  is  a  0.000* 

a.  On  a  peer  level  with  the  Geographic  Combatant  Command 

Commander 

b. Subordinate  to  the  Geographic  Combatant  Command  Commander 

c.  Senior  to  the  Geographic  Combatant  Command  Commander 

d. Attached  to  the  Geographic  Combatant  Command  Commander 

e. None  of  the  above 
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APPENDIX  B 

TASK  GUIDE  AUTHORING  TOOL  OVERVIEW 
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The  Task  Guide  software  system  enables  organizations  to  develop  sophisticated 
applications  called  procedures  that  retrieve,  compute,  update,  display,  and  accept  infonnation 
using  a  step-by-step  wizard-like  user  interface.  Task  Guide  provides  a  graphical  authoring  tool 
that  enables  authors  to  create  these  procedures  quickly  and  easily,  without  programming. 
Powerful  authoring  capabilities  enable  you  to  create  adaptive  procedures  that  detennine  which 
information  to  present  and  which  questions  to  ask,  depending  upon  the  situation.  Task  Guide  is 
modular  and  extensible:  it  can  be  embedded  within  a  larger  Java  applications  or  applet,  and  it  can 
embed  Java  user  interface  components  and  libraries. 

Task  Guide  procedures  can  be  used  to  implement  diverse  types  of  applications  such  as: 

•Training  tutorials  that  present  information  to  students,  pose  questions  or  problems,  and 
provide  tailored  hints  and  feedback.  Student  perfonnance  can  be  tracked  in  files  or  by  a 
SCORM  learning  management  system. 

•Debriefing  systems  that  analyze  and  discuss  the  student’s  perceptions,  decisions,  and 

actions  during  a  simulation-based  training  or  assessment  activity.  This  discussion  fonnat 
enables  the  software  to  refine  its  assessment  of  the  student’s  proficiencies,  provide 
instructional  feedback,  and  guide  reflection. 

•Perfonnance  support  systems  such  as  task  aids  and  decision  support  systems  that  guide 
users  through  analysis,  decision-making,  and  execution  tasks,  step-by-step. 

•Training  simulations  such  as  branching  scenarios  that  present  the  current  situation  in  each 
screen,  prompt  the  student  for  decision  or  actions,  and  branch  to  the  appropriate  next 
situation. 

•Adaptive  questionnaires  that  determine  which  questions  to  ask  based  on  previous  answers 
and  other  available  data. 


Flexible  branching,  looping,  and  calculation  logic  enables  Task  Guide  to  select  the 
appropriate  next  screen  and  generate  its  contents  dynamically,  based  on  the  user’s  inputs  and  the 
values  of  the  procedure’s  variables.  This  makes  it  possible  to  create  flexible,  dialog-like 
interactions  that  adapt  to  the  situation  and  user.  For  example.  Task  Guide  training  tutorials  can 
select  or  generate  different  hints,  feedback,  and  follow-up  questions  based  on  each  student’s 
correct  or  incorrect  answers  to  previous  questions  and,  optionally,  other  infonnation  about  the 
student.  Task  Guide  training  simulations  can  branch  to  the  next  situation  or  present  possible 
choices  based  on  the  student’s  previous  decisions  and  the  values  of  procedure’s  variables  used  to 
store  the  simulation  state.  Task  Guide  debriefing  systems  can  point  out  noteworthy  instances  of 
student  perfonnance,  assert  relevant  facts,  and  ask  probing  questions  about  the  student’s 
experience  and  perspective.  After  the  student  responds,  the  debriefing  systems  can  delve  more 
deeply  with  follow-up  facts  and  questions.  Task  Guide  perfonnance  support  systems  can  filter, 
calculate,  and  display  the  information  or  possible  options  that  are  relevant  to  the  cunent  situation 
more  succinctly,  so  users  can  assess  situations,  make  decisions,  or  execute  tasks  more  accurately 
and  efficiently.  A  single  Task  Guide  procedure  can  even  combine  different  types  of  interactions. 
For  example,  a  Task  Guide  procedure  could  provide  training  by  interleaving  tutorial  interactions, 
branching  scenarios,  and  debriefing. 
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Task  Guide  Training  Tutorials 

A  Task  Guide  training  tutorial  displays  a  series  of  screens  that: 

•Present  direct  (didactic)  instruction  to  students, 

•Pose  questions  and  accept  and  evaluate  each  student’s  response, 

•Provide  hints  and  feedback  on  each  student’s  responses, 

•Branch  to  the  appropriate  next  screen  based  on  each  student’s  response,  and 
•Tracks  the  student’s  responses  and  scores  using  files  or  a  SCORM  learning  management 
system. 

Task  Guide  procedures  can  use  response-specific  hints,  feedback,  and  branching  to 
provide  instruction  that  adapts  to  the  student’s  background  and  responses  during  the  tutorial. 
Task  Guide  procedures  can  pose  two  types  of  questions: 

•Text  input  questions  that  prompt  the  student  for  free-text  responses,  and 
•Multiple  choice  questions  that  prompt  the  student  to  select  one  or  more  choices. 

Task  Guide  provides  specialized  support  for  authoring  multiple  choice  items  that  include 
response-specific  hints  and  feedback. 


Task  Guide  Software  Suite 

The  Task  Guide  software  suite  provides  the  following  software  programs: 

•Task  Guide  Procedure  Editor  -  a  Java  application  used  by  authors  to  create  Task  Guide 
procedures, 

•Task  Guide  Procedure  Execution  Tool  -  a  Java  application  used  by  students  or  end  users  to 
run  Task  Guide  procedures,  and 

•Task  Guide  Procedure  Execution  Applet  -  a  Java  applet  that  runs  within  Java-enabled  web 
browsers  to  run  Task  Guide  procedures.  This  applet  is  invoked  by  embedding  an  applet 
tag  within  a  Hypertext  Markup  Language  (HTML)  web  page. 


Task  Guide 
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Editor 


Procedure  author 
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Task  Guide 
Procedure 
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Figure  12.  Authors  create  procedures  using  the  Procedure  Editor.  Users  run  procedures  using  the  Procedure 
Execution  Tool 
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Task  Guide  Procedures 


Task  Guide  procedures  are  comprised  of  step  nodes  and  group  nodes.  Step  nodes  specify 
the  content  and  format  of  each  screen,  along  with  optional  calculations  that  run  before  or  after 
each  screen  are  displayed.  Group  nodes  contain  step  nodes  and  lower-level  group  nodes.  Group 
nodes  organize  steps  within  a  hierarchy,  similar  to  the  way  computer  folders  organize  files.  The 
different  types  of  steps  and  groups  are  shown  below. 
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Each  simple  step  presents  infonnation  to  the  user  and  can  prompt  the  user  for  input.  An 
exit  step  specifies  an  exit  condition  that,  if  true,  instructs  Task  Guide  to  stop  executing  the 
current  group.  The  conditional  branches  step  specifies  many  possible  steps  to  execute  next, 
depending  upon  the  values  of  certain  variables.  A  simple  group  organizes  related  steps  and 
subgroups.  All  of  the  steps  and  sub-groups  contained  within  a  simple  group  are  executed  in 
sequence.  A  branching  group  specifies  a  Boolean  (true/false)  condition  that  determines  whether 
the  steps  in  the  group  should  be  executed  or  skipped.  A  looping  group  executes  its  steps 
repeatedly  while  its  test  condition  is  true. 

A  step’s  instructions  can  contain  input  controls  such  as  text  fields,  check  boxes,  radio 
buttons,  and  selection  lists  that  prompt  the  user  to  enter  data  and  decisions.  Task  Guide  stores 
user  input  values  in  variables,  so  they  can  be  used  within  the  calculations  and  test  conditions  of 
this  step  or  downstream  steps.  Task  Guide  provides  special  support  for  specifying  multiple 
choice  questions  with  response-specific  hints  and  feedback.  Task  Guide  steps  can  present 
verifications  that  tell  the  user  how  to  confirm  that  the  step  was  completed  successfully.  Optional 
notes  communicate  information,  cautions,  and  warnings.  Step  instructions  can  also  embed 
arbitrary  Java  graphical  user  interface  objects  that  provide  specialized  information  displays  or 
user-system  interactions. 

Steps  can  contain  calculations  that  evaluate  expressions  and  save  these  values  in 
variables.  These  expressions  can  contain  constant  values;  variables;  math,  text,  and  logical 
operations;  and  calls  to  Java  methods.  The  saved  variable  values  can  be  used  by  branching  logic 
to  control  which  downstream  steps  are  executed  and  which  are  skipped.  They  can  also  be  used 
within  calculations  in  downstream  steps  to  send/receive  data  to/from  other  systems  and 
databases,  analyze  and  interpret  this  data,  recommend  actions  to  be  taken  by  the  user,  or  select 
and  execute  actions  automatically.  Pre-calculations  execute  at  the  beginning  of  each  step  before 
the  step’s  instructions  are  displayed  to  the  user.  They  are  useful  for  retrieving  and  computing 
data  or  text  so  they  can  be  embedded  within  dynamic  instructions.  Post-calculations  execute  at 
the  end  of  the  step,  after  the  user  has  followed  the  step’s  instructions,  entered  data  into  input 
fields,  and  pressed  the  next  step  button  to  indicate  completion.  They  are  useful  for  interpreting, 
processing,  saving,  or  acting  upon  the  user’s  inputs. 
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Task  Guide  Authoring  Tool 


Authors  use  the  Task  Guide  Authoring  Tool  to  create  procedures.  A  Task  Guide 
procedure  encodes  step-by-step  instructions  and  execution  logic  as  a  list  of  steps,  organized 
within  groups  and  subgroups  in  a  hierarchy.  Each  step  presents  instructions  to  the  user  using 
formatted  text  and  graphics  specified  using  the  Hypertext  Markup  Language  (HTML). 
Instructions  can  contain  hyperlinks  to  web  pages  that  present  additional  infonnation  on  demand 
in  a  web  browser  using  text,  graphics,  and  other  media. 

The  Task  Guide  Editor  is  shown  below.  The  overview  pane  (left)  contains  tabbed 
windows  that  display  the  procedure’s  steps  and  groups,  along  with  the  variables  and  Java  library 
functions  that  can  be  used  in  procedure  steps.  The  details  pane  (right)  lets  authors  edit  the  step 
or  group  that  has  been  selected  in  the  left  pane.  The  bottom  pane  lists  text  search  results  and 
problems  detected  automatically  by  the  Editor. 


Figure  13.  Task  Guide  Procedure  Editor. 

Each  step  can  contain  static  or  dynamic  instructions  and  verifications.  Static  instructions 
present  the  same  information  each  time  the  procedure  specification  is  executed.  Authors  use  the 
built-in  graphical  HTML  editor  to  specify  the  content  and  fonnat  of  static  instructions  as 
formatted  text,  images,  and  hyperlinks. 

You  can  also  specify  dynamic  instructions  by  embedding  expressions  within  the 
instruction’s  HTML  text.  During  execution,  Task  Guide  evaluates  each  embedded  expression 
and  replacing  the  expression  with  its  value.  Expressions  often  contain  references  to  variables 
whose  values  can  be  entered  by  the  user,  received  from  external  systems  and  databases  during 
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procedure  execution,  or  computed  from  mathematical,  Boolean,  or  string  expressions  that  refer 
to  other  variables.  You  can  also  use  conditional  inclusion  to  include  some  text  and  exclude 
other  text  when  displaying  a  step,  based  on  the  values  of  certain  procedure  variables.  Compared 
to  static  instructions,  dynamically-generated  instructions  can  filter  information  to  present 
instructions  that  are  more  succinct  and  targeted  to  the  user  or  situation.  They  can  also  select  or 
compute  hints,  feedback,  default  values,  or  recommendations. 

Task  Guide  supports  gradual  automation,  so  manual  instructions  read  and  followed  by 
users  can  be  replaced  over  time  with  steps  that  use  calculations  to  retrieve  data,  compute  values, 
and  carry  out  actions  automatically.  For  example,  a  step  could  simply  provide  instructions  that 
tell  the  user  how  to  carry  out  a  particular  task.  This  step  could  be  enhanced  with  calculations 
that  automate  the  retrieval  of  relevant  data.  Further  enhancement  might  use  calculations  to 
compute  default  parameter  values  or  decisions  and  prompt  the  user  to  confirm  or  override  them. 
As  confidence  increases  in  the  reliability  and  robustness  of  the  automated  recommendations,  the 
organization  could  replace  the  step  with  a  fully  automated  step  that  analyzes  data,  decides  and 
acts  without  user  intervention.  In  this  manner,  a  manual  procedure  can  evolve  over  time  into  a 
more  automated  one. 

Task  Guide’s  extensible  architecture  enables  integration  with  general  purpose  and 
application-specific  Java  software  libraries  that  provide  functions  that  are  invoked  by 
calculations.  This  architecture  enables  Task  Guide  procedures  to  incorporate  sophisticated 
automated  data  retrieval,  interpretation,  automated  reasoning  and  decision-making  algorithms. 
The  Task  Guide  application  programming  interface  (API)  enables  you  to  embed  Task  Guide 
within  a  larger  Java  application  or  applet. 
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Appendix  C 

Authoring  Tool  Usability  Questionnaire  and  Responses 
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This  appendix  presents  responses  to  questions  on  the  usability  of  the  tutorial  authoring 
tool  provided  by  a  fonner  instructor  at  Joint  Forces  Staff  College. 


1. What  kind  of  changes/additions  do  you  think  an  instructor  is  likely  to  want  make  to  the 

Stability  Operations  Tutorials? 

•Correct/amplify/update  readings 
•Alter/correct/expand  questions  and/or  answers 

•Add/change  display  elements  to  reflect  newer  images/doctrine  or  enhanced  detail 
•Alter  reference  list 

For  each  of  these  four  areas,  the  primary  reason  for  executing  the  changes  would  be  to 
keep  current  with  evolving  doctrine  and/or  policy  statements  -  the  recent  natural  disasters  in 
Haiti  and  Chile  are  recent  evidence,  especially  when  one  considers  the  manner  of  response,  that 
stability  operations  are  a  constantly  changing/evolving  field  of  endeavor.  A  secondary  reason  for 
executing  changes  will  likely  be  to  adapt  to  either  the  specific  course  instructors  style  or  to  the 
variety  of  students  at  JFSC,  which  includes  foreign  officer  fellows  utilizing  English  as  a  second 
language. 

2.  Does  the  Task  Guide  authoring  tool  support  the  ability  to  make  those  changes? 

Yes,  the  authoring  tool  supports  making  all  of  the  above  listed  changes,  as  well  as  the  ability  to 
create  a  new  tutorial  from  scratch  -  given  the  instructional  design  prepared  by  the  applicable 
instructor. 

3.  Please  describe  your  assessment  of  the  usefulness  of  the  capabilities  provided  by  the  Task 
Guide  authoring  tool. 

Given  the  ability  to  substantially  alter  an  existing  tutorial  or  create  a  tutorial  from  scratch, 
I  find  the  authoring  tool  has  substantial  usefulness.  Among  the  capabilities  demonstrated  during 
the  training  period,  I  find  the  following  to  be  most  promising: 

•Ability  to  display  a  wide  variety  of  visual  elements  in  either  tutorial  or  exam  sections. 
•Ability  to  target  either  instruction  path  or  exam  paths  based  on  student  answers.  Evaluation 
of  student  answers  by  the  program  could  include  key-word  scoring  -  opening  up  the 
possibility  that  follow-on  interactions  would  be  able  to  fill  voids/amplify  weak  areas  vice 
cover  the  entirety  of  a  subject.  When  used  as  pre-instruction/preparation  for  a  lesson,  this 
would  allow  for  a  more  efficient  and  ultimately  effective  process  by  best  targeting 
individual  student  needs. 

4.  Please  describe  your  assessment  of  the  level  of  effort  needed  to  learn  the  tool  well  enough 
to  make  simple  changes  to  the  Tutorials.  Please  describe  what  you  assume  to  be  a  simple 
change. 
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Roughly  one  hour  of  review  of  the  supporting  documentation  and  2  14  hours  of  on-line 
collaboration  with  the  developer  were  necessary  to  prepare  to  make  the  changes  described  in  the 
answer  to  question  one.  The  changes  listed  in  the  answer  to  question  one  are  essentially  simple 
changes  -  editing/updating.  In  my  opinion,  the  level  of  training  necessary  to  accomplish  these 
simple  changes  will  by  necessity  include  sufficient  training  to  accomplish  more  complex  changes 
-  altering  question  structures,  adding  additional  instructional  paths,  etc.  This  results  from  the 
need  to  be  able  to  execute  changes  while  understanding  the  overall  structure  of  the  tutorial  - 
necessary  to  avoid  inadvertently  altering  the  structure. 

5.  Please  describe  your  assessment  of  the  level  of  effort  needed  to  learn  the  tool  well  enough 
to  make  more  complex  changes  to  the  Tutorials.  Please  describe  what  you  assume  to  be  a 
complex  change. 

I  would  estimate  that  an  additional  one  to  two  hours  of  on-line  collaboration  preceded  by 
an  additional  hour  of  study  of  the  operational  guides  would  be  necessary.  The  bulk  of  the 
collaboration  time  would  be  spent  in  monitored/guided  conduct  of  more  complex  changes.  The 
potential  intricacy  of  connections  should  in  my  opinion  be  laid  out  directly  in  front  of  the 
prospective  user,  ideally  through  at  least  two  to  three  examples.  By  complex  changes,  I  have  in 
mind  the  creation  of  a  tutorial  from  scratch,  the  addition  of  alternate  questioning/instructional 
paths  than  those  currently  existing,  or  the  addition  of  hints  and  retries  to  existing  questions  (not 
an  all-encompassing  list,  but  illustrative).  In  all  cases,  it  would  be  beneficial  to  reinforce  the 
essential  connection  between  an  instructional  design  and  the  specific  technical  tasks  being 
completed  -  the  power  of  the  tool  will  be  realized  best  if  the  effort  spent  at  instructional  design 
exceeds  that  of  the  time  required  to  make  the  technical  changes. 

6.  Please  describe  the  knowledge,  skills,  and/or  attitudes  required  by  instructors  to  use  the 
authoring  tool  successfully. 

Attention  to  detail,  advanced  preparation  of  changes  (to  include  flow-charting  of  more 
complex  changes),  detailed  knowledge  of  the  subject  matter  and  instructional  plan, 
documentation  of  changes  and  version  control,  basic  level  of  computer  familiarity  common 
among  instructors  who  utilize  education  technology  systems/networks  (the  tool  did  not  appear  to 
require  an  in-depth  knowledge  of  graphic  design  tools,  C++,  or  other  programming  languages). 

7.  Please  describe  strengths  of  the  Task  Guide  authoring  tool,  especially  those  that  you 
consider  to  be  relatively  unique. 

Graphic  interactions  that  appeared  to  be  easily  comprehended,  useful  and  well  indexed 
supporting  texts,  the  ability  to  test  changes  to  verify  operability  without  having  to  run  the  full 
tutorial,  the  ability  to  pursue  changes  at  the  level  of  detail  required. 

8.  Please  describe  changes  you  would  recommend  making  to  the  Task  Guide  authoring 
tool. 


When  pursuing  changes  to  some  of  the  question  structures,  the  display  included  the  base 
operating  code  language,  which  could  be  off-putting  to  some  less  programming-savvy  users. 

This  is  not  a  large  handicap,  but  interfaces  that  show  as  little  code  as  possible  and/or  simplify  the 
amount  of  manipulation  necessary  by  the  basic  user  would  be  beneficial. 
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9.  Can  you  identify  additional  training  applications  for  which  the  Task  Guide  authoring 
tool  might  be  an  appropriate  tool,  especially  those  that  would  be  useful  to  your 
organization?  If  so,  please  describe. 

One  potential  application  would  be  to  conduct  a  combination  pre-test  and  remediation 
process  for  incoming  students.  Arriving  students  can  have  very  different  levels  of  experience 
and  familiarity  with  the  concepts  of  joint  military  operations  and  planning  -  some  will  have  used 
them  extensively  and  recently,  some  will  have  been  instructed  in  them  at  a  point  several  years  in 
the  past  and  had  limited  opportunities  to  either  practice  or  refresh  their  knowledge.  The 
challenge  is  to  both  determine  this  level  in  advance  of  attendance  and  to  provide  the  students 
with  useful  feedback  and  recommendations  on  improving  their  familiarity.  I  envision  an  on-line 
pretest  that  would  support  multiple  tracks  based  on  the  individual  student’s  scores  and  familiarity 
with  “set-up”  questions,  and  then  guide  them  along  paths  that  address  weaknesses  while 
providing  opportunities  to  either  skim  or  bypass  areas  of  strength.  For  the  college,  these  results 
would  be  compiled  and  factor  into  the  assignment  of  students  to  seminars  to  produce  a  balanced 
level  of  expertise  and  an  improved  starting  level  for  the  course  of  instruction. 
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